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assessment portion of this effort. It utilizes and adapts an existing Model of 
Organizational Safety Effectiveness (MOSE) to achieve an understanding of the possible 
influences of organizational factors on aviation maintenance. This thesis develops and 
administers a prototype Maintenance Climate Assessment Survey (MCAS) that provides 
a tool for assessing safety in maintenance operations. The study has 268 participants 
from three Reserve squadrons that represent the spectrum of aviation communities. The 
prototype MCAS is comprised of 67 questions developed from 155 candidate questions. 
Each question uses a Likert type rating scale, which allows participants to express 
opinions for each item presented. Cluster and Factor analysis is used to identify 
redundancies between items and how items clustered according to the MOSE 
components. The product of this study 1s a finalized MCAS with 35 questions that can be 
used by the Squadron command and Aviation Safety Officer to assess their unit’s safety 
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EXECUTIVE SUMMARY 

Naval Aviation has been challenged to cut its 1996 human factors related Class A 
flight mishap rate in half by the year 2000. Numerous engineering and programmatic 
intervention strategies have been put in place to reduce aviation mishaps, and 
subsequently, the annual flight mishap rate. Improved aircraft reliability, advanced 
cockpit technologies, upgrades to avionics, and new acquisitions have lead to 
improvements that have helped to lower the mishap rate. However, mishaps are still 
occurring. Investigations show that while the mechanical-caused flight mishap rate has 
fallen significantly to an all time low, the human-caused flight mishap rate has not 
declined as rapidly and has possibly increased in the last 10 years (Naval Safety Center, 
1997). 

On the heels of the 29 January 1996 Nashville, Tennessee, F-14 fighter jet mishap 
that killed the aircrew and several local residents, Vice Admiral Bennitt, Commander 
Naval Air Force, U.S. Pacific Fleet, created a Human Factors Quality Management Board 
(HFQMB) to improve aviation safety by preventing human error. The HFQMB was 
chartered to examine factors contributing to human error, especially aircrew error. 

In 1997, the Navy and Marine Corps had an impressive year; their combined class 
A flight mishap rate was the second lowest ever recorded and it was the Navy’s best ever 
recorded (NSC, 1998). Given this success, a shift to improve maintenance was initiated. 
Of all the identified causal factors that contributed to Class A flight mishaps from fiscal 
year 1990 through 1997, 17 percent involved maintenance. “Maintenance” causal factors 
are primarily due to human errors by maintenance personnel and/or maintenance 


supervisors. The maintenance thrust of the HFQMB adapted the same approach of 
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benchmarking, mishap data analysis, and climate safety assessment. This thesis is the 
climate safety assessment portion of an ongoing study of Naval Aviation maintenance 
operations. 

Organizational culture has an effect on the entire organization. This is especially 
true in high-risk organizations where potential outcomes of a system failure can be 
catastrophic and costly. Naval Aviation is recognized as a high-risk organization, which 
can experience devastation if something goes awry. This also holds true for aviation 
maintenance aS well as operations. This study involves the analysis of data from a 
prototype survey that is based on an existing model of high-risk organizations. The 
purpose of this study is to develop, administer, and validate a Maintenance Climate 
Assessment Survey (MCAS) to assess the effectiveness of Naval Aviation maintenance 
operations in the management of risk. It utilizes and adapts an existing Model of 
Organizational Safety Effectiveness (MOSE). This study analyzes the MCAS data; the 
results are intended to improve our understanding of the possible influences of human 
factors in aviation maintenance related mishaps, and to provide a tool that can be used by 
the Squadron command and Aviation Safety Officer to assess their unit’s safety posture 
in conducting scheduled / unscheduled maintenance operations. 

The study includes a total of 268 participants, of whom 212 are maintainers, from 
three Reserve squadrons that represent a variety of Naval Aviation communities. The 
prototype MCAS comprised 67 questions developed from 155 candidate questions. Each 
question contained five point Likert scales (strongly disagree, disagree, neutral, agree, 
and strongly agree) which allowed participants to rate items according to their views. 


The questions fit into six components of the MOSE: Process Auditing, Reward System 
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and Safety Culture, Quality, Risk Management, Command and Control, and 
Communication / Relationships. The data analysis entails descriptive statistics, cluster 
analysis and factor analysis for objective identifications of redundancies among items and 
how items clustered according to the MOSE components. Redundant questions are 
eliminated and questions with common intent are combined. Some questions remain the 
same. Thorough analysis of the questions produce a finalized MCAS with a total of 35 
questions that can be used for individual squadron self-assessment of safety posture in 


scheduled / unscheduled maintenance operations. 
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I. INTRODUCTION 


Naval Aviation is an inherently complicated system, from flying the platform into a 
hostile environment to landing it on an aircraft carrier. Complex systems have many 
forces acting on them, some synergistically, some separately, all having either positive or 
negative effects (Perrow, 1986). Such forces and the effects they have on the system are 
difficult to grasp and understand (Bond, Byran, Rigney, & Warren, 1975). These forces 
can lead to pilot errors, cause failures in the platform or create a range of causal factors 
that can lead to the loss of a platform, or worse, lives, thus having a direct effect on 
combat readiness. Warfighting readiness and victory in combat can be achieved by 


keeping the platforms functional and the pilots alive (Nutwell & Sherman, 1997). 


A. BACKGROUND 


The Naval Aviation Safety Program (Chief of Naval Operations Instruction 
3750.6Q, 1989) was established to deal with aviation mishaps and any related causal 
factors. The purpose of this program is to preserve human and material resources while 
its Objective is to eliminate hazards, the causes of damage and injury. In essence, the 
program was set up to determine all hazards that can become causal factors leading to 
aviation mishaps. 

Numerous efforts have been employed in attempts to lower the Class A flight 
mishap rate (Figure 1). These efforts resulted in the creation of several institutions and 
programs, including the Naval Aviation Safety Center (now Naval Safety Center — NSC), 
the Naval Aviation Maintenance Program (NAMP), the Fleet Replacement Squadron 


(FRS) concept, the Naval Air Training & Operating Procedures & Standards (NATOPS), 


and the Squadron Safety Program (SSP). Additional efforts include Aircrew Coordination 


Training (ACT), 


Human Factors Councils/Boards (HFC/B), and Operational Risk 


Management (ORM). 
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Figure 1. FY 50-96 Flight Mishap Rates and Intervention Strategies. 
From ‘‘Naval Safety Center Brief,’ by Admiral F. Dirren, April 1997. 


All of the above efforts are intervention strategies put in place to reduce aviation 


mishaps and the annual flight mishap rate. The first five programs mentioned are design 


and programmatic in type. The last three break from an engineering and management 


focus to concentrate more on human factors issues. Improved aircraft reliability, 


advanced cockpit technologies, upgrades to avionics, and new acquisitions have lead to 


improvements that lower the mishap rate. However, mishaps are still occurring. Since 


1990, approximately 4 of every 5 Department of the Navy (DON) Class A flight mishaps 


(those involving aircraft damage in excess of one million dollars or fatal injury) involve 


human factors (Department of the Navy (DON), 1997). For example, in 1997, of the 27 


aircraft destroyed, 23 involved human factors. 


Naval Aviation has been challenged to cut its 1996 human factor related Class A 
flight mishap rate in half by the year 2000. To do this, it must systematically and 
continuously improve all of the processes, programs, and systems affecting the safety of 
Naval Aviation operations (Nutwell & Sherman, 1997). The Naval Aviation Class A 
flight mishap rate has been significantly reduced over the past few decades. However, it 


appears to have reached a plateau over the past decade as indicated in Figure 2. 
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Figure 2. Naval Aviation Class A Flight Mishap Trend FY 88-97. 
Rate is Defined as the Number of Class A, B, & C Flight Mishaps Per 100,000 Flight Hours. 


Though the Navy and Marine Corps flight mishap rate has generally decreased 
over the past two decades, the rate of decline has slowed. Investigations show that while 
the mechanical-caused flight mishap rate has fallen significantly to an all time low, the 
human-caused flight mishap rate has not declined as rapidly and is possibly leveling off 
or even increasing in recent years (NSC, 1997). Some conclude that this split is due to 
the complexity of the aircraft and current pressures of operations tempo, personnel 


downsizing, etc (Schmidt, 1998). Figure 3 shows this relationship. 
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Figure 3. Mechanical/Catastrophic Failures vs Human Errors Over All Mishaps. 
From ‘“‘Naval Safety Center Brief,’ by Admiral F. Dirren, April 1997. 


Figure 4 shows the percentage of determined causal factors that contributed to 
Class A flight mishaps from FY90 — FY97. Supervisory factors were present in 62 
percent of these flight mishaps. Examples of supervisory factors include inadequate 
command supervision of flight operations or safety, failures to correctly assess mission 
risks, and the inappropriate handling of a known high-risk situation. The aircrew was 
found to be a causal factor in 56 percent of the mishaps. Material factors, defined as 
material failure of the component, were present in 39 percent of the mishaps. 
Maintenance was found to be a causal factor in 17 percent of mishaps. This 
“maintenance” causal factor was primarily due to human errors by maintenance personnel 
Or maintenance supervisors. Facilities factors, relating to human errors in conjunction 


with airfield operations, contributed to five percent of the mishaps. 
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Figure 4. Navy and Marine Corps Fiscal Year 90-96 Class A Flight Mishap Causal Factors. 
From “Naval Safety Center Brief,’ by Admiral F. Dirren, April 1997. 


On the heels of the 29 January 1996 Nashville, Tennessee, F-14 fighter jet mishap 
that killed the aircrew and several local residents, Vice Admiral Bennitt, Commander ~ 
Naval Air Force, U.S. Pacific Fleet, created a Human Factors Quality Management Board 
(HFQMB) to improve aviation safety by preventing human error. The charter of the 
HFQMB is to analyze and improve each of the processes, programs, and systems that 
impact human performance in aviation with the purpose of dramatically reducing the 
annual flight mishap rate (DON, 1997). In other words, its main objective is to analyze 
human factors involvement in past Naval Aviation mishaps and in present Naval Aviation 
operations. Preventing a mishap contributes to readiness and mission success by keeping 
people and airplanes available through controlling safety-related hazards. Most of the 
HFQMB initiatives directly contribute to mission readiness, as well as indirectly, through 


mishap prevention (Nutwell & Sherman, 1997). 


Three processes define the HFQMB methodology: Benchmarking, Mishap Data- 
Analysis (MDA), and Command Safety Assessment (CSA). These lead to brainstorming 
and potential intervention strategies, which in turn result in recommendations presented 
to the Navy’s Air Board of Flag officers (Figure 5). Benchmarking occurred with visits 
to commercial airlines, private industry, sister services, NASA, the FAA, NATO partners, 
and academia. The objective was to examine the culture, training, safety, and leadership 
of each organization; to identify and learn about successful safety practices in order to 


adopt best practices, identify lessons learned, and avoid “‘re-inventing the wheel.” 





Figure 5. HEQMB Methodology. 
From “‘AIRPAC Brief,” by CDR J. Schmidt, March 1998. 


The MDA categorizes the human error elements of a mishap into three groups: 
unsafe supervisory conditions, unsafe aircrew conditions, and unsafe aircrew acts. The 
first two are seen as latent conditions and the last as active failures. Unsafe supervisory 
conditions include planned inappropriate operations, failure to correct known problems, 
supervisory violations, and inadequate supervision. Unsafe aircrew conditions pertain to 
aeromedical, personnel readiness and cockpit resource management (CRM) problems. 
Unsafe aircrew acts pertain to slips, lapses, mistakes, routine violations, and exceptional 


acts of the aircrew. Class A flight mishap data analysis from fiscal year 1990 through 
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1996 showed the following. Supervisory failure is a causal factor in 62 percent of Class 
A flight mishaps during fiscal year 1990 through 1996. Fifty one percent of the 
Supervisory failures are labeled as unsafe supervisory conditions. The unsafe aircrew 
events contained: 28 percent of the flight mishaps as aeromedical, 48 percent as CRM 
and three percent as readiness. The unsafe aircrew act event contained 87 percent as 
accidental and 40 percent as violations. (Nutwell & Sherman, 1997). Notably, in a report 
issued in 1997 by the DON, forty percent of the volations were conscious aircrew 
breeches of procedures, Standard Operating Procedures (SOP) and NATOPS. 

Ciavarelli’s and Figlock’s (1997) CSA study was conducted to assess a 
command’s safety climate from an aircrew perspective and the perceived effectiveness of 
its safety related programs. Its objective was to analyze the effectiveness of managing 
risks in flight operations. The study, based on a model of high reliability organizations, 
shows significant room for improvement in risk management, human factors evaluation, 
and aircrew coordination training. The study’s survey develops a tool] that provides 
commanding officers and aviation safety officers (ASO) a capability for self assessment 
of the effectiveness of their commands safe operations (Nutwell & Sherman, 1997). 

The HFQMB formed ten Process Action Teams (PAT) for areas that require 
intervention: Leadership, Policy, Organizational Effectiveness, Safety Information 
Management, Training and Qualifications, Standard Operating Procedures, Mishap 
Investigations, Operational Risk Management, Human Factors Evaluation and 
Aircraft/Aircrew Systems. The PAT were tasked to assess each area using the 


information taken from benchmarking, the MDA, and CSA, to identify area 


improvements, and recommend actions resulting in an awareness of human factors, causal 
factors of mishaps and, ultimately, to achieve a mishap rate reduction. 

To date, there has been little emphasis on the potential contribution of human 
factors on maintenance operations. The HFQMB’s first thrust was aimed at supervisory 
(62 percent) and aircrew (56 percent) causal factors due to the large percentage of 
mishaps involved, whereas discussed earlier, maintenance was found in 17 percent of the 
mishaps. In 1997, the Navy and Marine Corps had an impressive year; its class A flight 
mishap rate was the second lowest ever recorded and it was the Navy’s best ever recorded 
(NSC, 1998). Success in the intervention of these two areas has led to interest in 
developing efforts directed at the maintenance causal factors. 

The same three-pronged approach was adopted by the HFQMB for taking an in- | 
depth look into Naval Aviation maintenance operations for the identification of hazards 
and the development of intervention possibilities (Figure 5) (Schmidt, 1998). 
Benchmarking is currently underway and mishap data analysis is now completed. The 
benchmarking efforts have determined that risk management contributes to enhanced 
safety in terms of maintenance error prevention, lower mishaps, and reduced personal 
injuries. Organizational culture and climate, facilitation of crew coordination and team 
performance, effective training and certificate programs, and emphasis on policy and 
procedure adherence are all areas where risk management contributes to enhanced safety. 
These findings reflect some of the problem areas found in an analysis of Class A flight 
mishaps that involved maintenance depicted issues related to supervisory, crew 
coordination, training/qualifications, and adherence to policy and procedures. The 


results from benchmarking and mishap data analysis in conjunction with the previous 


CSA questionnaire were used to develop a candidate list of questions for a maintenance 


questionnaire. 


B. PURPOSE 


The purpose of this study is to develop, administer, and validate a questionnaire 
survey to assess the effectiveness of Naval Aviation maintenance operations in the 
management of risk. This study analyzes survey data and builds on a Model of 
Organizational Safety Effectiveness (MOSE) developed by Dr. Ciavarelli and LtCol 
Figlock (1996) of the School of Aviation Safety, Naval Postgraduate School, Monterey, 
CA. The results will improve our understanding of the possible influences of human 
factors in aviation maintenance related mishaps, and will provide a tool for assessing 
safety in maintenance operations. The intent is to identify potential interventions in an 


effort to lower human error in maintenance-related mishaps and the overall mishap rate. 


Cc: PROBLEM STATEMENT 


Given the sustained presence of human error in aviation maintenance, it is 
essential to develop a valid Maintenance Climate Assessment Survey targeting the 
maintainer’s perspective on safety and flight practices. Given this need, the existing CSA 
questionnaire based on the present MOSE must be adapted to address maintenance issues. 
It entails possible expansion and necessary modification of the existing questionnaire, 
which is geared towards aircrew, and the development of maintenance specific iene 
This thesis explores the following questions: 

1. Can the present MOSE be adapted to capture the maintainer’s perspective? 


2. Do the close-ended questions (questions with fixed responses) in the MCAS 
conform to the existing MOSE components? 
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3. Can the prototype MCAS be reduced through either the elimination of 
redundant questions or combination of such questions? 


4. Are there discernable differences between squadrons surveyed, and does there 
appear to be any consensus? 


5. Does the maintenance data support the components / structure of the 
maintenance adapted MOSE? 


D. SCOPE AND LIMITATIONS 


The intent of this study is to gain a better understanding of the Naval Aviation 
maintenance safety climate, understanding that efforts are underway to look into the 
Supervisory and aircrew factors. Three California based Reserve Squadrons are used as 
subjects in the survey due to their availability, accessibility, close proximity; this effort 1s 
a preliminary study. Neither Marine Corps units nor active duty personnel participated in 
this study, nor did any Tactical Air Squadrons. Only the Naval Aviation maintenance 


personnel responses are used in this analysis. Incomplete surveys are omitted. 
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IT. LITERATURE REVIEW 


A. BACKGROUND 


It has been evident for most of recorded history that people are prone to commit 


>? 


errors- “To err is human.” As our technology expands, as our wars multiply, and as we 
invade more and more of nature, we create systems - organizations, and the organization 
of organizations - that increase the risk for the operators, passengers, innocent bystanders, 
and for future generations. Most of these risky enterprises have catastrophic potential, the 
ability to take the lives of hundreds of people in one blow, or to shorten or cripple the 
lives of thousands or millions more (Perrow, 1984). Catastrophic errors in a system are 
often the result of the gradual aggregation of small errors across a system. Considering 
organizations as systems, and as parts of systems, is important in order to understand how 
organizations work, and how to best mitigate risk in them (Grabowski & Roberts, 1996). 


Advances in technology have increased the reliability of most system components, but the 


percentage of human error-related incidents and accidents has remained fairly constant. 


B. STUDY FRAMEWORK 


The theoretical framework for the construction of the maintenance questionnaire 
survey used in this study is derived from the work done by Ciavarelli and Figlock (1997) 
at the School of Aviation Safety, Naval Postgraduate School, Monterey, CA. The 
framework is based on key attributes of successful organizational risk management 
processes identified by Libuser (1994) and Roberts (1990), and from a safety climate 


analysis published by Zohar (1980). The existing CSA questionnaire was generated using 
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academic resources and professional questionnaires, which provided guidance regarding 
design and construction of questionnaires (Ciavarelli and Figlock, 1997). 

Five major areas concerning the effectiveness of organizations at managing risk 
are identified in a model developed by Libuser (1994). These areas included process 
auditing, a reward system, quality of operations, risk perception and command and 
control. Libuser’s work was adapted by Ciavarelli and Figlock (1997) to develop a 
Model of Safety Effectiveness (MOSE) that incorporates standard Naval aviation 
language and safety practices. The five MOSE components are summarized as follows: 


1. Process Auditing — a system of ongoing checks to identify a hazardous 
condition and a process to take corrective actions. 


2. Reward System — a system to recognize and reward safe behaviors, and to 
discourage unsafe behaviors. 


3. Quality standards — Sets control procedures to monitor quality and correct 
deviations to referenced standards. 


4. Risk Management — Systematic process to identify and manage nisk. 


5. Command and Control — Established policies and procedures to conduct safe 
operations. 


A complete description of the MOSE components is provided in Appendix A. 

Ciavarelli and Figlock’s (1997) short term goal is to provide the QMB with 
feedback on issues concerning aviation command climate, morale of Naval Aviation 
personnel, squadron workload and resource availability, the estimated success of ongoing 
safety interventions, and other factors related to safely managing Naval flight operations. 
Their long-term goal is to develop, validate and apply methods for assessing and 
mitigating risk in organizations that must, by the nature of their business, conduct 


hazardous operations. They develop and apply questionnaire survey methodology to 
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assess the effectiveness of Naval Aviation units in the management of risks associated 
with flight operations. This study is the first of several planned investigations intended to 
improve the understanding about possible influences of organizational factors in aircraft 
accidents, specifically understanding what possible influences a Naval Aviation 
command may have on the chain of events leading to an aircraft mishap (Ciavarelli & 


Figlock, 1997). 


Cc ORGANIZATIONAL CULTURE AND CLIMATE 


Roberts (1990) examines organizations that are successful in managing risks 
associated with hazardous operations. She labels this type of an organization as a “high- 
reliability” organization (HRO), which operates in hazardous environments yet produces 
a very low rate of accidents and incidents. Grabowski and Roberts (1996) contend that 
strong cultures and norms that reinforce their mission and goals characterize HROs. A 
culture is defined as the shared values, beliefs and assumptions that may govern decision 
making, attitudes, safety, and proper conduct within an organization. Norms are rules and 
regulations that guide and control the behavior of individuals and organizations. They 
focus attention on procedures, policies, and reward structures that are consistent with the 
organization’s mission. Grabowski and Roberts (1996) find that various cultures 
operating in large-scale systems are the glues tying the segments together. 

Zohar (1980) finds strong evidence that organizations with successful safety 
programs have a strong management commitment to safety. She determines that in low 
accident companies, top management is personally involved in safety activities on a 
routine basis, and safety matters are given high priority in company meetings and in 


production scheduling. This underlying safety climate refers to the shared perception of 
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an Organization’s members that the organization’s leaders are genuinely committed to 
safety of operations. She contends that such a climate results in increased performance 
reliability of workers, good housekeeping, and high design and maintenance standards for 


work environments. 


D. HRO: CLIMATE AND CULTURE 


To operate a complex system successfully, the human-machine system must be 
supported by an organizational infrastructure of operating concepts, rules, guidelines, and 
documents. In high-risk endeavors such as aircraft operations, it is essential that such 
support be flawless, as the price of deviations can be high. When operating rules are not 
adhered to, or the rules are inadequate for the task at hand, not only will the system’s 
goals be thwarted, but there may also be tragic human and material consequences (Degani 
& Wiener, 1994). Both Roberts (1990) and Libuser (1994) believe that organizations 
operating safely and effectively have certain key characteristics in common. Leadership 
style, sound safety management policies, procedure standardization, adequacy of 
resources and staffing, and a defined system for risk management are such characteristics. 
In large-scale systems, maintaining or developing a culture of safety and reliability can be 
important in mitigating mnsk. Oversight, checks and balances, and strong cultural norms 
are organizational protections and buffers that reinforce an organization’s goals; they can 
mitigate risk by assuring errors are caught, that appropriate organizational norms are 
developed and reinforced, and that the system improves over time (Grabowski & Roberts, 


1996). 
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E. AVIATION MAINTENANCE AND RAMP ACTIVITIES 


As anyone working in aviation maintenance knows, there have been a number of 
serious, even fatal, accidents over the years that were caused primarily by maintenance 
errors (Reason & Maddox, 1998). Boiled down to its essence, the task of a maintenance 
mechanic is to take off and then replace some of the three to four million removable parts 
on a modern aircraft platform. Much goes on in between, of course, but these basic steps 
of disassembly and re-assembly remain constant features of the work — the latter attracts 
by far the largest number of errors (Reason, 1990). The maintenance area is an unstable 
environment in a number of different ways: from the point of view of human factors it 
does not compromise a well-designed work environment. Problems range from complex 
activity and congestion in a poorly delineated environment to the problems of controlling 
the movement of large objects in space with inadequate information and cues (McDonald 
& Fuller, 1994). It is clear that problems with human factors still exist in aircraft 
maintenance facilities — even those that are built in strict accordance with the prevailing 
architectural standards (Maddox, 1998). Therefore, within maintenance activities, the 
human operator’s role is critical to safe and efficient operations. 

The technology of the ground handling process also poses interesting problems for 
safety. Aircraft are designed to fly; they are not optimally designed for ramp operations. 
The aircraft’s size and location in space relative to the operator invite misperceptions and 
misjudgments of distance and location; the fragile skin and appendages of an aircraft are 
easily damaged and such damage if undetected, can have disastrous consequences in 


flight. The cyclical nature of the ramp operator’s job often requires immediate and 
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demanding increases of both mental and physical performance from a resting and perhaps 
fatigued level (McDonald & Fuller, 1994). 

There are many factors within a maintenance and ramp activity that can be 
associated with failures (Reason & Maddox, 1998). These factors can have an adverse 
effect upon the working practices of those within the maintenance and ramp areas and 
hence, human performance. Factors within the workplace are known as local errors, and 
those that lie within the organization, organizational factors. The organizational factors 
create the local errors. 

One such local factor is the knowledge, skills and experience of maintenance 
personnel; personnel can be unfamiliar with a defect or aircraft type, lack specific training 
or skills and have inappropriate experience to perform proper maintenance. A morale 
factor can exist; people clash in personality, get frustrated, or can be unhappy with the 
work situation. Availability of tools, equipment and parts quality may be another factor. 
Other local factors may include fatigue, the pressures of a high workload, problems with 
shift patterns, and environmental problems. Unclear manuals and procedures, quality of 
safety equipment and training may also be present (Maddox, M, 1998). 

An organizational factor may be organization structures; there can be concerns 
about restructuring and downsizing, ill-defined duties, or too many layers of management 
in the organization. People management, the lack of top level awareness of problems that 
technicians face, and ill-defined career paths, may be an issue. Training, operational 
pressures, planning and scheduling problems and communication within the organization 
can also be examples of organizational factors. These factors are usually present within 


most maintenance facilities, and can directly impact the climate within which the 
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maintenance technician or ground handler must cope. The maintenance and ramp activity 
is a dangerous work environment presenting the risk of death or disabling injury to those 
involved. There is an ever-present potential for unreported ground handling damage to 
aircraft while in the hangar or on the ramp. Therefore, policies and procedures are 


needed to minimize this potential (Maddox, 1998). 


K. PHILSOPHY, POLICY, PROCEDURES AND PRACTICES 


Philosophy, policy, procedures and practices, the four P’s, are recognized as 
important aspects in flight deck operations (Degani & Wiener, 1994). Philosophy deals 
with the over-arching view of how the organization will conduct business, including 
flight operations. An organization’s philosophy is largely influenced by the 
organization’s leaders, and also by its culture. Policies are broad specifications of the 
manner in which management expects operations such as training, flying, and 
maintenance to be performed. An organization’s policy embodies the philosophy. 

Procedures should be designed to be as consistent as possible with the policies, 
which are consistent with the philosophy. Procedures exist to specify what the task is, 
when the task 1s conducted and by whom, how the task is done, and the sequence of 
actions needed to perform the task. Practices are an extension of and are governed by the 
philosophy, policies and procedures within an organization. A practice is the activity 
actually conducted on the flight deck and it is the ultimate factor that determines the 
quality of a systems output. The goal of management is to promote ‘good’ practices by 
specifying coherent procedures (Degani & Wiener, 1994). Maintenance, like flight deck 


Operations, is an activity that relies heavily on the four P’s described above; there is the 
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leadership philosophy, the policies within the maintenance department, the Standard 


Operating Procedures (SOP), and the technician’s or mechanics practices. 


G. ASSESSMENT AND INTERVENTION OF RISK 


Looking at all the components within the MOSE and seeing if these apply to the 
Naval Aviation maintenance community Is called its assessment. Do the five components 
of process auditing, reward system and safety culture, quality, risk management, and 
command and control relate to maintenance? Assessing maintenance operations shows 
that indeed, these components are present. There is a need for the auditing of 
maintenance logs and SOP. A strong safety culture is a must; everyone must be safety 
conscious and identify safety problems. Quality control is the heart of maintenance, 
maintenance is definitely a high-risk environment, and leadership must be present to 
ensure the commitment to safety. 

Intervention is the gathering of data from a process, the attempts to fit a model to 
the data, and learning more about the process. A better understanding of the 
organizational factors related to safely managing maintenance operations is needed and 
desired. A survey of the maintenance community, based on the MOSE adapted for 
maintenance, is one method of intervention. Surveys are most appropriate when 
information should come directly from people involved with the process. Surveys 
provide data describing attitudes, values, habits, and background characteristics such as 
age, education and income (Fink & Kosecoff, 1985). Cuiavarelli and Figlock’s (1997) 
MOSE can be adopted to incorporate maintenance terms and language, thus developing a 


needed tool for intervention. 
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H. SUMMARY 


Organizational culture has an effect on the entire organization. This is especially 
true in high-risk organizations where potential outcomes of a system failure can be 
catastrophic and costly. Naval Aviation is recognized as a high-risk organization, which 
can experience devastation if something goes awry. This is also true in the aviation 
maintenance community. Several approaches have been made to improve the overall 
safety of Naval Aviation: a system safety analysis, an analysis of mishap data and surveys 
to improve the understanding of organizational issues pertaining to operators. If the 
concepts of systems safety and organizational accidents are to be advanced, aviation 
management at all levels must be aware of them (Reason, 1990). There is a need to 
develop a survey to gain a better understanding of the organizational influences on 
maintenance error. In looking at different survey methods, a self-administered survey 


needs to be developed for the Naval Aviation maintenance community. 


bY 





It. METHODOLOGY 


A. RESEARCH APPROACH 


The intent of this study is to develop a Maintenance Climate Assessment Survey 
(MCAS) for the Naval Aviation Maintenance community that captures the maintainer’s 
perception and understanding of the nsks within their department and in performing their 
mission. This study involves the analysis of data from a prototype survey that is based on 
an existing model of high-risk organizations. This 1s done to identify factors that may 
improve safety within aviation maintenance. The data analysis entails descriptive 


statistics, cluster analysis and factor analysis. 


B. DATA COLLECTION 


1. Subjects 

A total of three Naval Air Reserve Force squadrons, two located at NAS North 
Island and one at Moffett FAF, are surveyed. The first is a Fixed Wing Fleet Logistics 
Suppett (VR) Squadron composed of active duty and selected reserve personnel 
providing seven-day-a-week round the clock, worldwide logistics to support the Navy and 
Marine Regular and Reserve forces. The second consists: of a Fixed Wing Maritime 
Patrol (VP) Squadron with a worldwide theater of operations, composed of Selected 
Reservists and active duty personnel (Note: This squadron is decommissioning).: The 
third squadron consists of a Rotary Wing Combat Support (HCS) Squadron with a 
mission of Search and Rescue, Combat Support via the launch and recovery of targets and 


torpedoes on the Southern California Offshore Range (SCORE). 
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The survey respondents are primarily Navy enlisted personnel who work in 
aviation maintenance departments. Maintenance personnel from the VR Squadron 
located at the Naval Air Station (NAS) North Island, CA were used to test the reliability 
and consistency of the prototype survey. The survey was then administered to two other 
groups of maintenance personnel from the VP and HCS Squadrons. 

2. Instrument 

The prototype MCAS consists of 15 demographic and 67 maintenance related 
questions. Demographic questions included inquiries about rank, community, shift 
worked, total years of service, and total years of aviation maintenance experience. They 
also include inquiries about unit home location, rating, age and current maintenance 
qualifications. The demographic questions preserve anonymity of particpants; names, 
social security numbers, work section, etc are not asked for. The MCAS is constructed 
using an existing air operations oriented Command Safety Assessment survey (CSAS) as 
a basis. 

Questions from the CSAS are re-worded to relate to “Maintainers” with 
maintenance terms. A focus group discussed all 156 candidate questions to reduce 
redundancy and ensure proper phrasing (Appendix B lists the candidate questions). 
Survey questions are fitted to the existing components as outlined in the current MOSE 
and new areas are created to fit any remaining questions. The MOSE components consist 
of Process Auditing, Reward Systems and Safety Culture, Quality, Risk Management, and 
Command & Control. Additional questions, proposed by current maintenance experts, 


are also incorporated into the MCAS. All questions are partitioned according to the 


22 


MOSE components, and the residual questions are examined. They fit under a sixth 
category peculiar to maintenance vice air operations: Communication / Relationships. 

The MCAS questions are separated into the MOSE components of Process 
Auditing, Reward System and Safety Culture, Quality, Risk Management, Command and 
Control, and Communication / Relationships. Eight of the 67 questions cover Process 
Auditing, 10 cover the Reward System and Safety Culture component, 12 cover Quality 
as well as Risk Management, 14 cover Command & Control and 11 cover the 
Communication / Relationships component. Eight of the 67 questions are worded 
negatively, where a positive response is indicated by a low value. The survey questions 
utilize a five point Likert rating scale with the following verbal anchors: Strongly 
Disagree, Moderately Disagree, Neutral, Moderately Agree, and Strongly Agree (The 
MCAS is included in Appendix C). 

3. Procedure 

The MCAS is distributed in person. All persons taking the MCAS are told what 
the survey is, why it is being given and told that their responses are completely 
anonymous. The Fleet Support Squadron received the MCAS during a safety stand 
down. Groups of 10 to 15 people were administered to until all available personnel had 
taken the survey. Each person was given approximately 15 to 20 minutes or more time if 
needed, to take the survey. The Patrol Squadron MCAS was given one on one going 
from office to office throughout the squadron’s workspaces over a drill weekend. The 
MCAS was handed out in a workspace and the participants were individually told what 
the survey was, why it was needed and that their individual responses would not be 


reported. The workspace MCAS took much longer to administer because participants 
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had a job to do. Some MCAS forms were completed right away, some were collected a 
few hours later. Throughout the workday and the workspaces, the MCAS were collected 
on an individual basis. The Combat Support Squadron personnel took the survey at the 
beginning of a safety brief over a drill weekend. The Squadron personnel were split into 
two groups of 30 to 40 people. Each group was given approximately 15 to 20 minutes or 
more time if needed, to take the survey. The MCAS was collected at the time it was 


completed. For each squadron, each survey was hand numbered for future order. 


C. DATA ANALYSIS 


1. Data Tabulation 

Survey demographics and responses are hand entered into an Excel (Microsoft, 
1996) spreadsheet. The spreadsheet consists of rows of respondents and columns of 
survey questions. The first fifteen columns are for dooce with another sixty- 
seven columns representing the actual survey questions. Results are coded in the 
spreadsheet by assigning scores of 1, 2, 3, 4, or 5 corresponding to the Likert scale of 
Strongly Disagree, Disagree, Neutral, Agree and Strongly Agree, respectively. Questions 
8, 20, 22, 39, 40, 41, and 54 are negative in wording and ask respondents to answer 
opposite to how they respond to the other questions. Survey questionnaire items that had 
no response are left blank and were dealt with by S-PLUS (Mathsoft, 1997) as the data 1s 
transformed into a SPLUS 4.0 data frame for complete data analysis. 

2. Statistical Analysis 

Basic summary Statistics are developed. Descriptive analysis is conducted on the 
data to describe basic and general information about the demographic and question 


results. These results include the distribution of survey participants by rank and service 
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classifications, and the total sample, mean and standard deviation for each of the 67 
survey questions. The S-Plus 4.0 (Mathsoft, 1997) program is used for multivariate 
Statistical analyses. Clustering methodologies such as agglomerative nesting and divisive 
analysis utilizing the S-PLUS functions AGNES and DIANA are used to answer the third 
research question with the aid of factor analysis. Factor and principal component 
analyses are performed to answer the second and fifth research questions, utilizing the S- 


PLUS functions FACTANAL and PRINCOMP. 
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IV. RESULTS 


A. DESCRIPTIVE STATISTICS 


Descriptive statistics of survey participants are presented in Table 1. For each 
participating Squadron, Fleet Support, Patrol and Combat Support, there are two 
columns. The column titled “All” is representative of all survey participants for the 
respective squadron. The column with the heading “Maint” pertains to maintenance 
respondents only. Table | lists the numbers of officers, E1/E3, E4/E6, and E7 and above 
for each squadron as well as the number of TAR, SELRES and USN designated 


personnel. 
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Table 1. Distribution of Survey Participants. 


From Table 1, the maintenance personnel involved in taking the survey represent 
79 percent of the total survey population. Maintenance E4/E6 enlisted personnel 
represented a total of 82 percent of the data analyzed in this thesis. This is important to 


note because most of the maintenance work and work supervision is done at this level. 
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B. MAINTENANCE RESPONSE STATISTICS 


The MOSE component results for maintenance respondents are presented in 
Tables 2 through 7 (The MOSE component results for all respondents are tabulated in 
Appendix D). Tables 2 through 7 list the maintenance results for all three Squadrons for 
each of the six MOSE components. The tables show total respondents for each question 
(n), average rating for each question on a “1” to “5” Likert scale and the standard 
deviation for each question. It is interesting to note that average results amongst the 
squadrons are relatively similar. Also noted is that for the eight questions of the MCAS 
worded negatively, the respondent’s answers reflect as expected. 

The first MOSE component presented in Table 2 is Process Auditing. There are 
eight questions in this component. The only question that raises a concern is number 59 
(overall mean = 2.73). All three Squadrons rate this question below neutral, in the 
disagree direction of the Likert scale. The question entails the use of safety staff to 
manage personnel at risk. A better, more effective use of safety staff and a higher 
presence of safety staff within the squadron may be a possible way to influence this 
perspective in the positive direction. The remaining eight questions are answered 
positively (mean range = 3.08 to 4.83). 

Fleet Support Patrol Combat Support Combined 

Squadron Squadron Squadron Squadrons 
| | Total | Avg [StdDev| Total | Avg [StdDev| Total | Avg |StdDev] Total | Avg |StdDev| 
pot | 86 | 4.42 | 0.458] 76 | 4.38 | 0.653 0.623 
2 | 86 | 3.79 | 0.897 | 76 | 3.79 | 0.838 0.952 
| 3 86 | 4.47 | 0.699 
4 86 | 4.41 | 0.879] 75 | 4.29 | 0.785 | 50_| 4.16 | 0.889 
0.562 
1.093 


. 


Table 2. Process Auditing MOSE Component. 
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The Reward System and Safety Culture MOSE component is presented in Table 
3. Ten questions comprise this component, with two that are negatively worded, 
questions 41 and 54. These two questions appear to be a problem, but reverse coding 
shows that they are answered positively (mean range = 3.49 to 4.02, and 3.02 to 3.45 
respectively). The other eight questions are all answered positively (mean range = 3.10 to 


4.32). 


peei"| "Squadron | squadron | "Squadron | Squadrons 
Squadron Squadron Squadron Squadrons 

| «Total | Avg |StdDev| Total | Avg |StdDev| Total | Avg |StdDev| Total | Avg |StdDev| 

PST 85] 3.51 | 1.158] 76 | 3.68 | 0.996} 50 | 3.82] 1.155] 211 | 3.67 | 1.103 | 

| 8 | 86 4.24 |o916 | 76 | 4.17 | 0.870] 50 | 3.98] 1.078 | 212 | 4.13 | 0.955 _ 

19 | 85 3.35 | 1.874 | 76 | 4.01 | 0.887] 50 | 4.08] 0.922] 211 | 3.82 | 1.228 | 

| 36 | 84 | 3.37 | 1.320] 74 | 3.20 | 1.007] 49 | 3.10] 1.141] 207 | 3.22 | 1.156 _ 
| 86} 2.51 | 1.406 | 76 | 2.24 | 1.118] 50 11.98] 1.040} 212 | 2.24 | 1.188 | 
| 86 _ | 3. 













| 3.64 | 1.198] 76 | 3.47 | 0.945] 50 [3.56] 1.091} 212 | 3.56 | 1.078 
| 60 | 85 | 4.31 [0.715 | 76 | 4.32 | 0.697] 50 | 4.32] 0.913] 211 | 4.31 | 0.775 | 


Table 3. Reward System and Safety Culture MOSE Component. 





The third MOSE component is Quality. This component has twelve questions, 
with one question, number twenty (mean range = 4.00 to 4.05), requiring reverse coding. 
All questions in this component, except one, are answered positively (mean range = 3.45 
to 4.37). Question 37 (overall mean = 2.82) is answered in a negative manner. This 
question relates to staffing; Is it sufficient from shift to shift. The Quality component 


results are presented in Table 4. 
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Squadron Squadron Squadron Squadrons 
| Ci «Total | Avg | StdDev| Total | Avg |[StdDev] Total | Avg {StdDev| Total | Avg |StdDev| 
F9 | 86 | 423 |os77{ 76 | 420 | 0654| 50 |440| 0.571] 212 | 4.28 | 0.601 
| 10 —C | 84 | 368 1 1.160] 75 | 361 | 0.769] 50 | 3.70] 0953] 209 | 3.66 | 0.961 | 
| 17 | 86 | 4.29 | 0.632] 75 | 4.20 | 0.838] 50 | 4.28[0.858| 211 | 4.26 | 0.776 | 
20 86 | 197 le 280 76 eS 044 oo OC 22 BT ager eo | 
| a7 | 84 | 263 [1.344] 75 | 3.07 | 0.991] 50 | 2.76] 1.205] 209 | 2.82 | 1.180 | 
| 42 | 86 S| 3.45 [1.757 | 74 | 3.80 | 0979] 50 | 3.80] 1.143 | 210 | 3.68 | 1.293 | 
| 44 | 86 | 4.24 10.987 | 75 | 4.28 [0.745] 50 | 4.26/ 1.006} 211 | 4.26 | 0.913 | 


Table 4. Quality MOSE Component. 


Risk Management is the fourth MOSE component and it has twleve questions, 
with three questions, numbers 12 (mean range = 2.84 to 3.03), 22 (mean range = 3.55 to 
3.88), and 40 (mean range = 2.93 to 3.12), requiring reverse coding. All but one of the 
questions were answered positively (mean range = 2.98 to 4.33). Question 51 (overall 
mean = 2.78) has negative responses. This question concerns equal workloads and equal 
stress / fatigue between shifts. This negative perception may be of some concern. The 


Risk Management component results are presented in Table 5. 


ee | seule | scien | | Sei 
Squadron Squadron Squadron Squadrons 
[| Total | Avg [StdDev| Total [ Avg [StdDev| Total | Avg [StdDev| Total | Avg |StdDev| 
pad | 84 | 2.98 11.180] 74 | 3.00 | 0.828] 50 | 3.24] 0.847 | 208 | 3.07 | 0.952 | 
Pee Peo) saa 7ord Or Menicerercen on a\pennercon ate | mcom ee ecoo 
2 
4 










2 

3 
4 | 86 | 

| 40 | 86 | 3.07 | 1619] 74 | 288 | 0859{ 50 | 2.88] 1.223] 210 | 2.94 | 1.234 | 


Co 


| 85 | 3.99 | 1.107 | 76 | 4.15 | o.706| 50 | 4.12] 0.961] 211 | 4.08 | 0.925 | 
| 62} 84 | 4.12 fo925 | 76 | 4.20 | 0.783] 50 | 4.28] 0.784 | 210 | 4.20 | 0.831_ 
1.437 | 76 | 3.95 | 1.018] 50 | 4.14] 0.948 


Table 5. Risk Management MOSE Component. 


Command and Control is the fifth component of the MOSE model. It comprises 






the largest number (14) of questions in the survey, with question 39 worded negatively. 
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Reverse coding question 39 (mean range =2.54 to 2.92) shows that it 1s responded to 
negatively in the survey. Question 39 deals with passdown between shifts within the 
command. All other questions in this component were answered positively (mean range 
= 2.96 to 4.68). Table 6 tabulates the results for the maintenance survey for the 


Command and Control MOSE component. 


a Squadron Squadron Squadron Squadrons 

| | Total | Avg |StdDev} Total | Avg [StdDev| Total | Avg |StdDev| Total | Avg |StdDev| 
J i386 4.4i | 0.621] 76 | 4.28 | 0.759 | 50 | 432] 0.713 | 212 | 4.33 | 0.698 | 
[25 | 86 | 400 | 0918] 76 | 4.05 [0.798] 50 | 4.08| 0.900| 212 | 4.04 | 0.872, 
| 26 | 86 | 3.90 | 1.036] 76 | 4.13 | 0.660] 50 | 4.04] 0.968 | 212 | 4.02 | 0.888 | 
| 27 | 86 =| 3.91 | 0.932] 75 | 4.08 | 0.749] 50 | 3.92} 0.966} 211 | 3.97 | 0.882 | 
| 28 | 86 | 3.94 | 0.950] 76 | 3.95 | 0.798] 50 | 3.90] 0.886} 212 | 3.93 | 0.878 | 
p29 86 | 3.71 [i733] 75 | 3.27 | 1212] 50 | 3.76] 1.153 | 211 | 3.58 | 1.179 | 
| 30 86 =| 3.81 | 1.259] 76 | 4.08 | 0.707 | 50 | 4.06] 0.998 | 212 | 3.98 | 0.988 | 
| 64 85 | 4.01 | 0.964] 76 | 4.00 | 0.766] 50 | 416] 0.842 | 211 | 4.06 | 0.857 | . 
66 | 83 2.96 | 1.206] 75 | 3.07 | 0.920] 50 | 3.14} 0.990] 208 | 3.06 | 1.039 | 


Table 6. Command and Control MOSE Component. 

























The final component of the MOSE is Communication / Relationships. This 
component has eleven questions, with question eight (mean range = 3.23 to 3.70) worded 
negatively. Questions 34 (overall mean =2.99) and 43 (overall mean = 2.64) show some 
concern, resulting in response values less than three. The Communication / Relationship 


component is tabled in Table 7. 
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Fleet Support Patrol Combat Support Combined 
Squadron Squadron Squadron Squadrons 





|_| Total 
| 8 84 | 77 [1575] 75_ | 2.39 | 1.102] 50 | 2.30] 1.165 | 209 | 2.49 | 1.281 | 
| 15 | 86 | 3.63 | 1.531] 76 | 3.33 | 1.038] 50 _ | 3.28] 1.161 | 212 | 3.41 | 1.243 | 


| 3st 86 | 3.73 | 1.257] 76 | 3.40 | 1.034 { 50 | 3.52] 1.216 | 212 | 3.55 | 1.169 | 
| 33 | 86 | 3.92 | 0.970] 75 | 3.96 | 0.892 | 50 | 3.94] 0.998 | 211 | 3.94 | 0.953 | 
| 384 | 86 | 2.90 | 0.918] 73 | 3.16 | 1.028] 50 | 290] 1.055 | 209 | 2.99 | 1.000 | 
| 52. | 86 | 3.79 | 1.203] 75 | 3.99 | 0.762] 50 | 3.92| 0.900} 211 | 3.90 | 0.955 | 


Table 7. Communication / Relationship MOSE Component. 


Each question within the MOSE is answered positively for all participating 
squadrons. The average for all questions in each MOSE component was computed and Is 


tabulated in Table 8. 


Squadron’ | Squadron Squadron Squadrons 


Table 8. MOSE Component Summary Results. 












c. ANALYSIS PROBLEM 


The dataset used in this portion of the analysis uses only the Maintenance surveys. 
Of the total 268 surveys collected, 212 were from Maintainers. These 212 responses are 
used to form a data matrix, X, to be studied and analyzed using the S-Plus statistical 
package. If X is n by k, then n = 212, the total number of survey respondents and k = 67, 
the number of survey questions presented. 

The problem is to reduce the existing k MCAS items to a more reasonable number 


for a full-scale survey. An acceptable number of questions for a survey is in the range of 
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25 to 40 questions (Fink, 1985). Clustering methods such as agglomerative nesting and 
divisive analysis are sensible tools for such problems. 

These methods belong to the family of hierarchical clustering methods that 
produce a tree-like hierarchy. This structure is most conveniently visualized through a 
dendrogram such as the one depicted in Figure 6. The vertical scale is a distance measure 
that marks the levels at which two clusters are joined to make one. The horizontal is a 
nominal scale on which the item designators are permuted so that the tree arcs do not 
cross one another. From the bottom up we can view how the k items are joined to form 
successive clusters. The use of such information can help provide a dispassionate guide 


for grouping the 67 items into a more reasonably sized set. 


D. AGGLOMERATIVE NESTING (AGNES) 


Two S-PLUS algorithms are used, AGNES and DIANA. Agglomerative nesting 
(AGNES) is discussed first as it is the easier to describe. It is a “bottom up” approach so, 
initially, there are (k=67) clusters; each single item is viewed as a cluster. First, one 
selects a distance measure. Suppose euclidian distance is chosen and is initially 
computed as the distance between each pair of questions. One finds the smallest value of 
this set and forms a cluster of the two items associated with it. Now, there are k-1 
clusters instead of k, and the distance of all items from the new cluster of two must be 
computed and used to replace the 2 * (k-2) distances of the two selected items from all 
others that have been supplanted. They are needed no more but there is a choice in 
defining the new distances. It was chosen to average the distances between all pairs of 


items, one from each cluster, to obtain a distance figure separating two clusters. This 
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done, again the smallest distance is chosen for forming the next new cluster, etc. The 
process continues until there is but one cluster consisting of all k items. The user can 
select the clustering solution that suits him/her. Scanning the vertical scale is often useful 
for this purpose. 

The agglomerative nesting function called “AGNES” in S-PLUS package is used 
as a first method to cluster the k questions or variables of the Maintenance safety survey 
into fewer questions. The S-PLUS function “AGNES” is called using the command 
agenes(daisy (t(X)),diss=T) . The S-PLUS function “daisy” is used to calculate 
the dissimilarity matrix for X. The “diss=T”’ statement means that a dissimilarity matrix 
is to be used in the computations to serve as the distance function. This choice 
corrresponds to the one described. 

Results from AGNES are then plotted using the pltree.agnes S-PLUS function. 
The pltree.agnes function produces a clustering tree of agglomerative hierarchical 
clustering when it is passed an AGNES object. It creates a plot of a clustering tree. The 
leaves of the tree are the original objects or in this case, the k questions. Two branches of 
the tree come together at the distance between the two clusters being merged. Figure 6 


shows the clustering tree resulting from the pltree.agnes command. 
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Figure 6. AGNES Clustering Tree. 


The AGNES function returns an object containing the following items: order, 
height, agglomerative coefficient, and a matrix called merge. Order is a vector giving a 
permutation of the original objects to allow for plotting, in the sense that the branches of 
a clustering tree will not cross. Height is a vector with the distances between merging 
clusters at the successive stages. The agglomerative coefficient (ac) measures the amount 
of clustering structure found in the dataset. More explicitly to each item 1, denoted by 
m(i), the dissimilarity to the first cluster it is merged with, is divided by the dissimilarity 
of the merger in the final step of the algorithm. The ac value is the average of all the {1- 
m(i)}. The resulting ac value produced by AGNES is 0.5445873. The AGNES object 
has a resulting merge matrix, describing the merging of a at steps within the 


clustering process. It provides the numerical support for the dendrogram. 


5) 


Merge is a (k-1) by 2 matrix, where each row 1 of merge describes the split at step 
k-1 of the clustering. The merge item returned using AGNES is important. This matrix is 
a road map for reading the clustering tree plot resulting from the pltree.agnes function. 
This matrix and the tree plot are used as an aid in clustering the questionnaire items. 
First, pairs are clustered utilizing the interpretations of the merge matrix. Next, triple and 
quad clusters are identified. Easily identified pairings, triples and quads are shown in the 
Table 9. Also shown in Table 9 are the singles, eight questions that do not merge with 


previously constructed clusters until high in the tree. 








| Singles | —SsPairings S| SCTriples |, Quads _ 
[66,67 





Table 9. AGNES Clustering. 


From these identified pairings, additional items were then added by using the 
corresponding merge information until all questions were covered. Table 10 shows the 
remaining questions and how they were added to the initial clusters. It shows the 


questions, identified by the AGNES algorithm, that are near in distance. 


1,5,9,13,17,47,56,60,62 11,34,36,66,67 46,48 37,51 
6,10,15,16,23,29, 31,38 8,20,22,41,54 49,50 39,40 
30,33) |(44,45,57,58,59,61,64,65)| (18,21,24,25,26,27,28)| (4255) | | 


Table 10. Additional AGNES Clustering. 






i. DIVISIVE ANALYSIS (DIANA) 


The AGNES algorithm is a bottom up approach to hierarchical clustering. The 
DIANA algorithm reverses this process. It is top down and the resulting tree need not be 


the same. The distance between clusters is used for explanatory purposes. 
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The process begins with a single cluster of all items and then considers all 
possible partitions of the items into two clusters. (It is convenient to think of it in this 
way, although efficient algorithms exist which do not actually do this.) The pair of 
clusters selected is the one with the greatest separation. Then the process 1s repeated using 
each of the clusters so formed, etc. Ultimately, the cluster will consist entirely of single 
items. The dendrogram for this method appears in Figure 7. Comparing Figure 6 and 
Figure 7 it is seen that the trees are quite similar. This is a comforting check. 

The divisive analysis function called “DIANA” in the SPLUS package is used as 
an approach to “cluster” the k questions into a tree. The DIANA function in S-PLUS is 
called using diana (daisy(t(X)),diss=T), where X represents the data matrix 
used in AGNES. Again, the S-PLUS function “daisy” is used to calculate the 
dissimilarity matrix for X. The “diss=T” statement means that the same dissimilarity 
matrix is to be used in the distance computations. 

Results from DIANA are then plotted using the pltree.diana function to look at the 
clustering results. The S-PLUS function pltree.diana creates a plot of a clustering tree or 
dendrogram given a DIANA object. The leaves of the tree lead to the k original objects or 
questions. A branch splits up at the diameter of the cluster being split. Figure 7 shows the 


resulting tree of “clusters” from the DIANA analysis. 
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Figure 7. DIANA Clustering Tree. 


The DIANA function returns the following items: “order’, “height”, the “divisive 
coefficient’, and “merge”. Order is a vector giving a permutation of the original objects to 
allow for plotting, in the sense that the branches of a clustering tree will not cross. Height 
is a vector with the diameters of the clusters prior to splitting. The divisive coefficient 
measures the amount of clustering found. More explicitly, for each item 1, denoted by 
d(i), the diameter of the last cluster to which it belonged ( prior to being split as a single 
object ), divided by the diameter of the whole set. The dc measure is the average of all 
{1-d(i)}. The divisive coefficient resulting from DIANA is found to be 0.6559955. Like 
the AGNES object, the DIANA object has a resulting merge matrix, describing the 
merging of clusters at steps within the clustering process. 

Again, the merge matrix and the tree plot are used as an aid in clustering the 


questionnaire items. First, pairs clustered are identified and analyzed to see if they make 
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sense. Next, triple and quad clusters are identified. Easily identified pairing are shown in 


the Table 11. 











gles | C—“‘“C;C*é@Patirings Triples | Quads 
12,40,43 | 11,34,37,51 
18,24 37,51 57,58 | 10,16,38 | 20,22,41 | 19,42,55,53 
25,26,27,28 













25,26 | 44,45 44,45,46,48 
27,28 66,67 61,62,64,65 


Table 11 DIANA Clustering. 


From these identified pairings, additional items are then added using the 





corresponding merge information until all questions are covered. Table 12 shows the 


remaining questions and where they are added to the initial pairings. It shows the 


questions, identified by the DIANA algorithm, that are near in distance. 


(1,5,9,13,47,56,60) | (57,58,59,61,62,63,64,65) | (8,20,22,41,54) | (6,10,16,23,38)| (3,4) 
(11,12,34,37,40,43,57) | (15,29,31,35,36,39,66,67)} (30,32,33,52) (2,7,14) (49,50) 





Table 12. Additional Clustering with DIANA. 


Comparing the two S-PLUS methods of AGNES and DIANA shows some 
interesting aspects. First, by comparing the two clustering trees, the shape of each tree is 
very similar. The initial pairings found in the two methods are alike also. Of the 23 
initial pairings from DIANA and the 21 pairings from AGNES, 17 are common to both. 
Moving up the clustering tree one level, more similarities are seen. Furthermore, by 
looking at higher levels of the two clustering trees, clusters of three or four questions 
form. Table 13 shows the common pairings, triples and quads found between the two 


methods. 
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F. 


questionnaire size reduction process. Principal Component Analysis was conducted on 
the dataset using the S-PLUS software package. The S-PLUS command of 
princomp(X,scores=T,cor=T) was used in the analysis. 
survey responses and “cor=T”’ means that the principle component analysis is to be based 
on the correlation matrix. The resulting principle component output was plotted using a 
screeplot, a special S-PLUS barplot function for the class princomp. The S-PLUS 


function screeplot(prin2.out) was used. It shows the contribution of each factor to the 












Table 13. Common Clusters Between DIANA and AGNES. 


PRINCIPAL COMPONENTS / FACTOR ANALYSIS 


The Principal Component / Factor Analysis material will help complete the 


total variance. Figure 8 shows the resulting screeplot. 
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X 1s the 212 by 67 matrix of 
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Figure 8. Principal Component Screeplot. 


Above each bar in the plot are the cumulative fractions of the variance. The 
variance is sharply the highest on the first component and spread rather evenly over the 
remaining components. This shows evidence that the survey data is dominated by one 
dimension. Much of the variability is captured in the first component. This and Factor 
Analysis is used to assist in answering research questions three and five. 

Factor analysis on the data is performed using S-PLUS. The S-PLUS command 
of factanal(X,factors=6,method=’ principal’, rotation=”,varimax’’) was used. Again, X is 
the 212 by 67 matrix of survey responses, six factors were fit, using varimax rotation as 
the method of aligning the factors in a six dimensional space. The “factors=6” input 
specifies the number of factors in the solution. An object of class factanal results. 

The factanal class results from fitting factor analysis models. This class has 
certain properties that are relevant to analyze. An important property is “loadings.” 


Loadings is an orthogonal matrix that gives the loadings for each factor, each column 
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being linear combinations of the columns of X for the corresponding factor. Another 
property of the factanal class is uniqueness. The variance of each item is decomposed 
into the part that is common to the factor solutions and the part that is unique to that item. 
The loadings and uniquenesses for each question in the corresponding components of the 


MOSE are presented in Tables 14 through 19. 


|Question| Factor 1 | Factor 2 | Factor 3 | Factor 4 | Factor 5 | Factor 6 [Uniqueness 
pt 0.462 | 0.306 | 0.149 | ove | 0.6381 __ 
Pp 2 | 0.337 | 0.278 | 0189 | 0219 | 0263 | | 0.6553 
p38 | 0.525 | 0.281 | -0.133 | 0.102 | 0.268 | -0.147 | 0.5263 
p 4 | 0.452 | 0.214 | 0.131 | 0.230 | -0.140 | 0.6555 _ 
| 5 | 0.478 | 0107 | | ey |} 0.195 | 0.6862 _ 
ee el a ee a me ee 
| 59 | 0.234 | 0.596 | 0.243 | 0.133 | 0148 | | 0.4885 
| 67 | 0.143 | 0.339 | 0436 | | 0152 | T0506 


Table 14. Factor Loadings for Process Auditing Component. 

















Coin . 

7 | 0.401 | 0.287 | 0.180 | 0.267 | 0.322 | | 0.5456 

| 18 | 0.459 | 0.313 | 0.297 | 0.384 | 0.447 

p19 | 0.366 || 0.128 | 0139 | 08288 
0.314 | oni7 | 0.329 | 0.441 | -0.160 | 0.5576 __ 


| Question | Factor 1 | Factor 2 | Factor 3 | Factor 4 | Factor 5 | Factor 6 {Uniqueness 
| 0.273, | 0.216 | 0.378 | 0.207 | 0.338 | | 0.5693 __ 
9 


_ 0.239 | 0.131 | | 0.202 | 0.277_ | -0.105 | 0.7925 _| 
_-0.179 | -0.188 | | 0.372 | 0.165 | 0.7532 _| 
| 0.249 | 0.593 | 0.152 | 0.310 | 0149 | | 0.4392 

0.625 0.204 0.133 


Table 15. Factor Loadings for Reward and Safety Culture. 


| 0.402 | -0.213 | 0.116 | -0.253 | | -0.455 {| 0.5007 _ 
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(Question | Factor 1 | Factor2 | Factor | Factor 4 | Factor5 | Factor 6 [Uniqueness 
[9 | 0507 | 0207 | 0200 | | 0297 | | 0.5898 
[10 [0.440] 0.147 | 0.373 | | 0.284 | -0.100 | 0.5573 
-—17_ [0617 | 0221 | | 0.193 | 0.250 | 0.101 | 0.4796 _ 
30] 0.222 | -0.327 | 0.116 | 0258 | | -0.464 | 0.5402 
[~a7_| | 0.118 | 0587] | 0.184 | 0.121 | 0.6469 
[42 0276 | 0147 | 0412 | 0127 | | 0245 | 0.6564 _ 
4a] 0.520 | 0239 | 0.124 | 0.189 | | 0181 | 0.5830 
[45 | 0.469 | 0.159 | 0154 | | | 0.152 | 0.6991 
Bar| ocsosnimees | |. | | oss 
[a8] 0567 | 0115 | 0461 | 01 | | -~+|~0.4258 
[49] 0.425 | 0215 | 0.238 | 0.250 | 0214 || 0.6017 
[50] 0.401 | 0.333] 0.410 | 0205 | | 0.752 | 0.4908 


Table 16. Factor Loadings for Quality. 






|Question| Factor 1 | Factor 2 | Factor 3 | Factor 4 | Factor 5 | Factor 6 |Uniqueness 
9 | 0343 | | 0.114 | -0.275 | 0.7637 _ 
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a 2 | a |e Oo Poses o.5236) 
Patt 0572 | 0.158 | fT 0.226 | 0.256 | 0.5284 
| 22 | 0.243 | -0.146 | | 0.203 | 0.473 | 0.6092 _| 
| 23 | 0.162 | 0.252 | 0149 | | 0.363 | 0.144 | 0.7275 | 
24 | 0487 | 033 | T0415 | 94 0.4371 
a | | eee | 2 ee | ee eeeer edamame Or77alz | 
SS ee ee ee Oe ee a ee 
| st | 0.239 | 0.638 | 0.321 | 0.157 | 0.176 | | 0.3709 


Table 17. Factor Loadings for Risk Management. 

















|Question | Factor 1 | Factor 2 | Factor 3 | Factor 4 | Factor 5 | Factor 6 |Uniqueness 
| 13 | 0.483 | 0.240 | | 0.158 | 0.272 | 0.183 | 0.5747 
0.4384 
| 0.328 | 0.338 | 0.258 | 0609 | | 0.3318 
27 | 0.417 | 0.155 | 0.218 | 0.642 | 0.166 | | 0.3063 
| 28 | 0.376 | 0.327 | 0.206 | 0.563 | 0.147 | | 0.3690 
| 0.338 | 0.198 | 0.213 | 0.176 | 0.523 | | 0.4972 


0.5029 


pC 0.454 | 0.131 | 0.5032 

I Se eT re 
0.5305 
| 0.238 | 0.609 | 0.211 | 0.231 | 0.193 | | 0.4305 
| 0.210 | 0.587 | 0.387 | 0.141 | 0.103 | | _0.4293 
0.579 | | 0122 | ~~ | 0.5808 


Table 18. Factor Loadings for Command and Control. 
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Question | Factor 1 | Factor 2 | Factor 3 | Factor 4 | Factor 5 | Factor 6 [Uniqueness 
8 | -0.160 | -0.166 | -0.107 | -0.268 | -0.101 | -0.295 | 0.7663 
p15 | 0.136 | 0.264 | 0.337 | 0.159 | 0.521 | 0.120 | 0.4876 
76 | 0.449 | 02637 | 0.199 | | 04a 0.4908 
| 31 0.206 | 0.226 | 0.248 | 0.279 | 0621 [| 0.3735 
| 32 | 0.339 | 0.142 | 0.252 | 0.307 | 0.122 | | 0.6877 
33 | 0.336 _ | 0.417 | 0.156 | 0.372 | 0191 | | 0.5109 
p34 | 0122 | 0.232 | 0.320 | 0.317 | -0.145 | 0.7043 
sss 0300 See Ea eee 
iit ae 


ee Oe 
ll ar a 0166 | {0.9669 
L562 | 0.441 | 0.181 | 0.308 | 0.208 | | 0.322 


0.4701 


Table 19. Factor Loadings for Commuinication / Relationships. 


For each component, the questions having the maximum loading over the six 
factors are analyzed. As true in the principle component analysis, there 1s one factor, 
Factor one, which had the most loadings having the maximum value. This analysis, with 
results shown 1n Table 20, shows the relationships between the factors resulting from the 
factor analysis and the MOSE components. Of the 67 questions, 25 of them load the 
highest in factor one. This provides quantitative information on these items that are 


represented in the single most important factor dimension. 














| Factori | Factor 2|Factor3| Factor4 | FactorS | Factor 6 
[Component1] 12,345.46 | 59 | 67 
[Component2] —7,18,19 | 5860 | 6 | 54 | 86,53 a 
[Component 3| 9,10,17,44,45,47,48,49] 4. 37,42,50] 
[Component4{ 21,24 61,62,63]11,5155] ae 


eS 

12,22,40 
56,64,65| 39,66 | 25,26,27,28 29,38 Ci a 
16,32,52 eg.o7 15,16,31,34,35,43| 8 | 


Table 20. Questions and Factors Loadings Broken Out by MOSE Component. 





Table 21 shows the distribution of the questions over the Factors and MOSE 
components resulting from the factor analysis. Interesting to note is the number of 


questions that belong to factor 1. Twenty five of the 67 questions in the MCAS fall into 


the first factor. The remaining factors have loadings with various signs and provide 


contrast type information about the items that dominate them. 


| Factor 5 | Factor 6 [Component Totals 
_ 


ae 
ee 
Component $73 





Table 21. Distribution of Questions From Factor Analysis. 


Also interesting to study is the cumulative distribution function of the 
uniquenesses for each question resulting from the factor analysis. This is important to 
study because the questions with low uniqueness in the large, have their content 
represented by other questions. The questions with high uniqueness have individual 
dominating status. From this, one may be able to identify questions that can be 
eliminated because they may be non-contibutors. Also, we can identify questions of high 
uniqueness, which either should be retained as singletons, or identified as dominating the 
group to which they belong. Figure 9 is the resulting S-PLUS CDFDISC function plot of 
the sorted uniquenesses and associated probabilities (Appendix E shows the S-PLUS 


CDFDISC function code). 
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Figure 9. Question Uniqueness Empirical CDF. 
Looking at the plot above gives a good view of the uniqueness for the questions. 
Note the flat spot appearing in Figure 9 at the 0.4 uniqueness level on the x-axis. The 
questions relating to these uniqueness values are possible candidates for elimination since 


they all have relatively low uniqueness. Below is Table 22 showing the seven questions 


whose uniqueness fall near or below the 0.4 value. 


Question Uniqueness 
re Ee 
| 27 | (0.3063 
28 | 0.3690 __ 
ee Xe 
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Table 22. Questions with Low Uniquenesses. 


There is also a flat area Figure 9 at the 0.75 uniqueness level on the axis 
identifying questions with high uniqueness relative to the entire set. All uniquenesses to 
the right of 0.75 relate to those having a relatively high uniqueness. There are ten such 


questions. Table 23 identifies the ten questions showing the uniqueness values for each. 
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Table 23. Questions with High Uniqueness. 


All of this information is useful to aid the researcher in making informative 
decisions about the items in the data. Each of the above areas of analysis are used in 
conjunction with one another to obtain the goal of a reduced set of questions to be used in 


an improved and modified MCAS. 


G. QUESTION ANALYSIS 


The S-Plus functions AGNES, DIANA, PRINCOMP and FACTANAL assist the 
researcher in making decisions or objective choices about the items. These will be 
viewed for reasonableness based on subjective knowledge. The stability of these choices 
and the results from the analysis are also influenced by the relatively small sample size 
involved in this study. With this in mind, the questions are studied using the computer- 
aided output from the previous sections. The prototype MCAS had questions that 
required reverse coding. It is recommended these questions be rephrasedto read in a 
positive aspect. (Appendix H lists these questions and the recommended modifications.) 

Decision rules need to be identified as to how the downsizing of the MCAS is 
approached. First, the seven questions identified with low uniqueness are removed. 
Likewise, questions identified, as having high uniqueness, will be retained. Secondly, all 
pairs that are the same in both AGNES and DIANA are studied for combination or to see 
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if the questions can be reworded to make one out of the two. This done, the next step is 
to take all triples and then quadruples identified by the clustering algorithms and look for 
sensible combinations or removal of questions. At the completion of this, the remaining 
questions that have not been grouped or clustered will be studied for further sensible 
reduction. 

Questions having low uniqueness are eliminated. There are seven such questions 
that have been identified in Table 22. The elimination of these seven questions results in 
a remaining pool of 60 questions. Furthermore, we retain the ten questions identified in 
Table 23 that have high uniqueness. This leaves 50 questions for continued analysis. 
Next, the cluster analysis results from AGNES and DIANA are used for combining 
questions that are answered in similar ways or show redundancy. 

Questions paired up that are common to both the AGNES and DIANA methods 
are analyzed to see if the pairings can be combined easily. Here, questions are paired up 
due to their relative closeness resulting from the computer-aided output. Questions are 
studied and if sensible, can be joined with the conjunction “and” or reworded to capture 
the intent of the original questions. For example, the first pairing from Table 13 is the 
pair (9,13). By rewording these questions slightly, a new question results that replaces 
the two questions with one. 

This process is done for all identified clusters in Table 13 as well as possible 
clusters from Tables 9 and 11. Table 24 lists these groupings. Appendix F gives the 
suggested rewording of these newly formed questions to be included in the suggested 


MCAS. 
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Nineteen questions are suggested as candidates for combining, rewording or 
grouping that reduce the MCAS question total by an additional 24 questions. A sensible 
pairing is made from two questions having high uniqueness. DIANA identifies pair 
(19,53) as a cluster and AGNES shows agreement in that the dendrogram has the pairing 
in close proximity to one another. This pair belongs to the same MOSE component and 
the two questions have the same intent, thus the two are combined into one. Question 11 


also had high uniqueness but it was sensible to cluster it as noted in Table 24. 


1,913 |11,36,59,67} 20,22 
10,66 30,33 52,56 
16,38 57,58 


18,21 ,24 37,51 64,65 
19,53 45,46,48 


Table 24. Suggested Question Groupings. 





Nine singletons result in this process. These questions neither clustered with 
others nor seemed reasonable to combine with any other question in the MCAS. These 


singletons are presented below in Table 25. 





Table 25. Original Questions Remaining. 


By combing the results in the first step, deleting the questions with relatively low 
uniqueness and then combining questions using the cluster analysis results, the MCAS 
reduces to 35 questions; seven from the high uniqueness, 19 from the clustering or 
rewording, and an additional nine as singletons. There is a judgement call here, and 
given the data and the questions, this seems to be as far as one can go without adding 


additional scrutiny to the study. Table 26 gives the suggested mapping of the 67-question 
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prototype MCAS to a suggested 35-question MCAS. The P-Type column contains the 
ordinal number of the item in the original 67-question MCAS, and the Mapped column 


displays the ordinal number in the suggested 35-question MCAS. 


| P-Type| Mapped | P-Type| Mapped | P-Type| Mapped | P-Type | Mapped | P-Type| Mapped | P-Type| Mapped | P-Type| Mapped | 
pf 8 tS et ts St Deleted | 41 1 _25 1 _St_ 22 | 8t_| Deleted 
pot ft 2 | oT 22 ft 15 32 Deleted | 42 | 26 | 52 | 31] 62_ | Deleted | 

| 33 


ie 06 2a ed ae ae 
ptf 4 fa fe far) peleted | a7 fee a7 ef te Te 
| 29 dT se | 33 
eS Se Some seco an Pee 
SA Ss 





Table 26. Prototype MCAS Questions Mapped to Suggested MCAS Questions. 
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V. CONCLUSION 


A. FINDINGS 


The existing MOSE can be used to model or capture the maintainer’s perspective, 
with modifications presented in this thesis. The close-ended questions, after tailoring 
them towards the maintainer, conformed to the existing MOSE components quite readily. 
However, an additional component, Communication / Relationships, was added to make 
the MOSE maintenance specific. Results of the prototype MCAS were positive in nature 
for all six of the adapted MOSE components except for a few questions that raised 
concern and these are listed in Table 27. These questions may possibly identify areas that 
need strengthening within the squadrons. 

Fleet Support Patrol Combat Support Combined 
Squadron Squadron Squadron Squadrons 


Table 27. Questions That Raise Concern. 


Question 











59 
37 
51 
39 
34 
43 





There are no discernable differences identified between the squadrons that were 
surveyed. A consenus between all participating squadrons was that of favorable results 
for all MOSE components. Agglomerative nesting and Divisive Analysis or enenvith 
Prinicpal Components and Factor Analysis were tools used to identify clusters of 
questions that were similar. Although the Factor Analysis shows that most of the 
questions load on factor one, there is no evidence against the structure of the maintenance 
adapted MOSE. Redundant questions were eliminated and questions with similar intent 


were combined. Thus, the prototype MCAS was reduced from 67 to 35 questions, 
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shortening the survey to a more practical length, requiring less time to complete. The 


finalized MCAS 1s listed in Appendix I. 


B. OTHER AREAS OF CONCERN 


There are numerous future studies that can further this thesis. One such task 
would be to alter the questionnaire in order to use it with the Marine Corps, using 
“Marine” specific terms. Also, this study could be further investigated by surveying 
active duty Naval Aviation Squadrons to make comparisons of the results with the 
Reserve Squadron results presented in this study. The Marine Squadron results could be 
compared to Navy Squadrons to see if there is a statistical difference between the two. 
Another task would be to administer the survey to Regular active duty Squadrons and do 
a comparison between Reserve and active duty Squadrons. Also, other analysis methods 


could be used to validate this study and its results. 


c. SUMMARY 


The Class A flight mishap rate has leveled off over the past decade. Human 
Factors studies are increasing as they tyy to identify additional areas where intervention 
could lead to further reductions of the Class A flight mishap rate. This thesis is part of a 
Human Factors related study specifically aimed at Maintenance operations within Naval 
Aviation. The resulting survey is to be used as a tool for commanders to assess the 


effectiveness of Naval Aviation maintenance operations in the management of risk. 
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APPENDIX A. EXISTING MOSE COMPONENTS 


COMPONENT 1: PROCESS AUDITING 


ee ae 


Conducts periodic safety reviews and inspections. 

Conducts timely reviews and updates of safety practices and operating standards. 

Uses a systematic process to set training goals and to review flight qualifications. 
Uses Human Factors Councils and Boards to identify and screen high-risk aviators. 
Makes effective use of Flight Surgeon for assessing significant stress reactions, and 
possible unsafe attitudes of individuals as well as routine medical screening. 


COMPONENT 2: REWARD SYSTEM 


ly 


am 


Recognizes safety achievement through social praise, and formal awards and 
incentives. 

Safe behavior is reinforced as a cultural norm (by command emphasis and peer 
pressure). 

Takes timely action (s) to appropriately discipline unsafe behavior and attitudes. 
Encourages everyone to be safety conscious and to identify safety problems. 
Encourages everyone to report safety discrepancies without fear of negative 
repercussions. 


COMPONENT 3: QUALITY 


a ee 


Sets high quality standards as an organizational goal. 

Publishes quality standards and quality control procedures. 

Monitors quality and corrects deviations from established quality standards. 

Gains reputation for high-quality performance in comparison to reference standards or 
comparable organizations. 


COMPONENT 4: RISK MANAGEMENT 


AR WN = 


ON 


Has accurate perception of actual operational mission risks. 

Uses a systematic method for managing risk. 

Risks of mission operations are acknowledged and minimized. 

Safety risk decisions are made at the proper level (by most qualified people). 
Resources (time, budget, staffing, and equipment) are adequate for performance of 
mission and organizational workload. 

Does not compromise safety to get the work done or to accomplish the mission. 


53 


COMPONENT 5: COMMAND AND CONTROL 
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Supervisors openly demonstrate a commitment to safety. 

Supervisors clearly communicate safety goals and policies. 

Supervisors are directly involved in safety management. 

Safety is considered as an integral part of mission accomplishment. 

The command establishes and uniformly enforces operating standards. 

There is frequent interaction between command supervisors and staff. 

There is good communication flow up and down the command chain. 

Safety training is emphasized throughout the command. 

The status and respect for the safety officer's position are high. 

Personnel turnover is not high enough to affect a command's ability to maintain 
"corporate" knowledge and the core cultural values of the organization. 

Supervisors provide adequate guidance and counseling on safety matters. 

Formal rules and procedures are followed, whether or not command supervisors are 
physically present. 

There is redundancy, multiple coverage or adequate safety backups for high-risk 
operations. 

Personnel in the command conduct continuous proficiency training. 

Training incorporates safety guidelines and safeguards. 

Leadership has vision and understands how to create a positive climate. 


.The prevailing values, beliefs, attitudes, and norms (culture) promote safe 


behavior/discourages unsafe behavior. 

Leadership sets example for written policy, standards, proper procedures, and 
acceptable norms of behavior. 

The command clearly establishes responsibility and accountability, for safe flight 
operations at all levels. 

Command leaders have an accurate perception of the motivation, morale, and job 
satisfaction level of their people. 

Command leadership reacts well, and readily adapts to unexpected changes. 


PROPOSED COMPONENT 1: PROCESS AUDITING 


0 > Oe SS ee 


My command adequately reviews and updates safety practices. 

The command has a dedicated program that targets individual training deficiencies. 
My command monitors maintainer qualifications. 

Support equipment licensing 1s monitored in this command. 

Tool control is taken seriously at my command. 

CDIs/QARs routinely monitor maintenance evolutions. 

My command uses safety staff to manage personne] at risk. 

The command uses medical staff to manage occupational hazards and personnel at 
risk. 
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PROPOSED COMPONENT 2: REWARD SYSTEM AND SAFETY CULTURE 


Ihe 


.' > ao 
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My command recognizes individual safety achievement through rewards and 
incentives. 

Unprofessional behavior is not tolerated in the maintenance department. 

Supervisors encourage reporting safety concerns without fear of retribution. 
Supervisors discourage violations of SOPs, or NAMP guidelines. 

My MO/MCPO understands if I feel uncomfortable performing maintenance duties 
due to personal issues. 

Violations of SOP, NAMP guidelines, or other procedures are common in my 
command. 

Peer influence discourages violations of SOP, NAMP guidelines or other procedures. 
Personnel are uncomfortable telling supervisors about personal problems including 
illness. 

Individuals feel free to report safety violations, unsafe performance, or other unsafe 
behavior. 


10. Our command climate promotes safe maintenance and flight operations. 


PROPOSED COMPONENT 3: QUALITY 


0 Cog ON Ose ee 


My command has established standards and maintains quality control. 
CDIs/QARs are sought after positions in my command. 

Inspectors perform all required actions before sign off. 

To meet operational commitments, supervisors allow “cutting corners.” 
Maintainer staffing is sufficient from shift to shift. 

Proper tools and equipment are available, servicable and used. 
Required publications are available, current and used. 

Maintenance gripes are either corrected or addressed prior to flight. 

My command has a reputation for quality maintenance. 


10. The QA division 1s respected in my command. 
11. Signing off PQS/JQRs/PARs is taken seriously and not gun decked. 
12. Maintenance quality on detachments is the same as that in homeport. 


PROPOSED COMPONENT 4: RISK MANAGEMENT 


SO ee 


My command temporarily restricts maintainers who are having personal problems. 
Based upon my command’s current manning and assets, it 1s over-committed. 
Supervisors manage hazards associated with maintenance and flight line operations. 
Supervisors are more concerned with mission completion than aircraft maintenance. 
My division CPO 1s aware of individual daily workload requirements. 

Unsafe conditions are recognized and addressed by M/C, Q/A, or W/C supervisors. 
Personnel turnover negatively affects my command’s ability to operate safely. 

Day and night check have equal workload and are equally stressful/fatiguing. 

I am provided adequate resources (time, personnel, and equipment ) to accomplish my 
job. 


5S 
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Safety decisions are made at the proper command levels. 

Safety 1s part of maintenance planning, and additional training/support is provided as 
needed. 

Maintainers are never purposely put in an unsafe situation to meet the flight schedule. 


PROPOSED COMPONENT 5: COMMAND AND CONTROL 
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1. In my command, we believe safety is an integral part of all maintenance and flight 


My command ensures all maintainers are responsible and accountable for safe 
maintenance. 

My command ensures the uniform enforcement of SOPs among unit maintenance 
personnel. 

Supervisors communicate command safety goals, programs, and procedures. 
Supervisors are actively involved in the safety program and management of safety 
matters. 

Supervisors set the example for compliance to established maintenance standards. 
Supervisors are responsive to unexpected changes and anticipate potential hazards. 
W/C supervisors are respected by the maintenance chief/officer. 

All maintenance evolutions are properly supervised by qualified personnel. 
Maintenance control is effective in managing all maintenance activities. 

Multiple job assignments and collateral duties adversely affect maintenance. 


line operations. 

Safety education and training in my command are comprehensive and effective. 
The safety department is respected by supervisors and maintainers. 
Maintenance Safety Petty Officer is a sought after billet in my command. 


PROPOSED COMPONENT 6: COMMUNICATION / RELATIONSHIPS 
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My command has a problem with passdown between shifts. 

Within my unit, good communication flow exists up and down the chain of command. 
Coordination is conducted between the M/C, W/C and QA prior to incorporation of 
TDs. 

Work center supervisors, division CPOs and M/C work well together. 

Aircraft moves are briefed and detailed personnel are qualified. 

Maintainers are briefed on potential hazards associated with maintenance activities. 
My supervisor shields me from outside pressures, which may affect my work. 

QARs are never pressured by the maintenance supervisors to sign off a gripe. 
Maintenance Control never troubleshoots aircraft discrepancies. 


10. QARs are viewed as helpful, and QA is not “feared” in my command. 


Ly 


I feel I get all information (internal and external) required to perform my job safely. 
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APPENDIX B. CANDIDATE MAINTENANCE QUESTIONS 


My command has a reputation for high quality maintenance. 

My command is genuinely concerned about safety. 

Command leadership is successful in communicating its safety goals to unit 
personnel. 

My command provided a positive climate that promotes safe maintenance and flight 
operations. 

Command leaderships is actively involved in the safety program and management of 
safety matter. 

Command leadership sets the example for compliance to established standards. 

My command conducts adequate reviews and updates of safety practices. 

My command ensures that all unit members are responsible and accountable for safe 
maintenance and flight operations. 

Command leadership permits “cutting corners” to get the job done. 


. Safety decisions are made at the proper levels by most qualified people in my 


command. 

Command leadership encourages reporting safety discrepancies without fear of 
negative repercussions. 

Individuals in my command are willing to report information regarding safety 
violations, marginal performance, or other unsafe behavior. 

My command has a defined process to set training goals and to review performance. 
Loss of experienced personnel has negatively affected my command’s ability to 
operate safely. 

Command leadership willingly assists in providing advice concerning safety matters. 
My command has established quality standards and strives to maintain quality control. 
Command leadership considers safety issues during the formation and execution of 
operational and training plans. 

Command leadership has a clear picture of the risks associated with its maintenance 
and flight line operations. 

My command takes the time to identify and assess risks associated with its 
maintenance and flight line operations. 

My command does a good job managing risks associated with its maintenance and 
flight line operations. 

My command closely monitors proficiency and currency standards to ensure 
maintenance personnel are qualified to perform their assigned tasks. 

My command provides adequate safety backups to catch possible human error during 
hi-risk tasks. 

Command leadership reacts well to unexpected changes to its plans. 

In my command, we believe safety is an integral part of al maintenance and flight line 
operations. 

My command has increased the chances of a mishap due to inadequate or incorrect 
risk assessment. 
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26. My command does not hesitate to temporarily restrict from maintenance tasks 
individuals who are under high personal stress. 

27. 1am adequately trained to safely conduct all of my assigned tasks. 

28.I am provided adequate resources (time, staffing, budget, and equipment) to 
accomplish my job. 

29. No equivalent. 

30. My collateral duties adversely affect my ability to accomplish my maintenance tasks. 

31. Morale and motivation in my command are high. 

32. My command ensures the uniform enforcement of operating standards among unit 
personnel. 

33. [have adequate time to prepare for my assigned tasks. 

34. Command leadership is effective at discouraging violations of operating procedures, 
NAMP guidelines or general maintenance discipline. 

35. In my command, peer influence is effective at discouraging violations of operating 
procedures, NAMP guidelines, or general maintenance discipline. 

36. My command’s SOP is effective at promoting safe maintenance and flight line 
operations. 

37.1 am very familiar with the policies and regulations contained in the OPNAVINST 
4790.2 series. 

38. Rest standards are enforced in my command. 

39. Based upon my command’s personnel and other assets, the command is over 
committed. 

40.In my command, PC/CDI/QAR tests and monitors are conducted as intended, to 
candidly assess maintainer qualifications. 

41. Strict enforcement of NAMP standards is upheld in my command. 

42.In my command, anyone who intentionally violates NAMP/NAMSOP 1s swiftly 
corrected. 

43. In my command, violations of operating procedures, NAMP regulations, or general 
maintenance or flight line operation discipline are rare. 

44. Within my command, good communication flow exists up and down the chain of 
command. 

45. My command has good two-way communication with external commands. 

46. Procedures in my command are adequate to effectively conduct Human Factors 
Councils or Boards. 

47. Human Factors councils have been successful in identifying maintenance members 
who pose a risk to safety. 

48. Human Factors Boards have been successful in managing the high-risk maintainer. 

49. My command makes effective use of the flight surgeon to help identify and manage 
high-risk personnel. 

50. My command recognizes an individual’s safety achievements through rewards and 
incentives. 

51. Safety education and training are adequate in my command. 

52. The safety department is a well-respected element of my command. 

53. The Maintenance Safety Petty Officer position is a sought after billet in my command. 
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My commands safety department keeps me well informed regarding important safety 
information. 

My command has received adequate guidance, information, and training regarding 
Maintenance Risk Management. 

My command has begun to implement MRM processes into decision-making at all 
levels. 

The QA division is well respected in my command. 

CDI is a sought after position in my command. 

QAR is a sought after position in my command. 

The EAWS program is considered to have integrity in my command. 

A coordinated brief is conducted between M/C, the W/C and QA prior to 
incorporating the AFB/AFC’s, etc. 

CDI/QARs perform routine monitors of maintenance evolutions. 

Requirements for PPE use are strictly enforced. 


Decisions on who should be CDI/QAR are made fairly and without favoritism. 
Support equipment licensing is closely monitored. 

Requirements for having SE licenses are taken seriously. 

The “signing-off’ of PQS/JQRs/PARs, etc. is taken seriously and managed with 
integrity. (not gun decked) 

Maintenance Control provides for adequate time to complete the job when assigning 
tasks. 

Performing incomplete daily inspections is not tolerated in my command. 

Inspectors never sign off the “inspected by” block without physically performing the 
required inspection. 


. Work center supervisors work well with M/C. 
i. 
74. 
1 
TS 
TE 


W/C supervisors are respected by maintenance management in my command. 

W/C supervisors are respected by the “troops” in my command. 

W/C supervisors are always asked their opinion by M/C. 

QA is a “feared” organization in my command. 

All participants involved in an aircraft move are briefed prior to commencing the 
move. 

All maintenance evolutions are well supervised. 

Maintenance evolutions are OVER supervised. 

Anyone in my command can stop a maintenance evolution if they feel that an unsafe 
condition exists. 

All pilots strictly follow plane captain directions/signals. 

Work center shift passdowns are conducted face to face. 

Daily maintenance meetings encourage feedback from work centers. 

All work centers utilize a written passdown log. 


. Division CPOs ensure 100% participation at monthly maintenance safety meetings. 
. Workers with “bad attitudes” are not permitted to work on or around aircraft or other 


equipment. 
Utilizing someone’s NALCOMIS password 1s strictly prohibited. 
Maintenance Control is ALWAYS in control of maintenance. 
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So. 
90. 
ils 
92. 
93: 
94. 
oD. 
96. 
oT 


98. 


ao. 


100. 
101. 
1023 
103. 
104. 
105. 
106. 
107. 
108. 


I have confidence that all personnel are qualified to perform their assigned tasks. 
“Down” gripes are taken seriously by QA. 

“Down” gripes are taken seriously by M/C. 

Repeat gripes are monitored by QA and regarded as serious. 

Publications are always used when performing maintenance. 

Maintenance Control never troubleshoots from behind the M/C counter. 

Plane captains always carry daily decks when performing daily inpsections. 

A799s are taken seriously. 

Maintenance training is conducted regularly and taken seriously by maintenance 
management. 

CPO’s are regarded as the technical experts and get involved in troubleshooting tough 
gripes. 

CPO’s over supervise maintenance tasks. 

Tool Control is taken seriously at my command. 

The maintenance control chief is respected. 

My CO is proud of the maintenance department. 

Squadron pilots feel comfortable flying the aircraft in this command. 

Our maintenance department is considered to be the best in the wing. 

Short cuts are tolerated when the flight schedule dictates. 

The maintenance department works for the operations department. 

The goals of the maintenance department are clearly defined. 

Command leadership is more concerned with mission completion that aircraft 


material conditions. 


109. PPE is available for use when needed. 

110. Most maintenance is performed by NX. 

111. NX maintainers are considered to be proficient. 

112. NX maintainers are considered to be as professional as DX. 

113. “Khaki presence” is evident at this command. 

114. NX is well represented with CPO’s. 

115. Shortcuts are tolerated more at night than DX. 

116. Working nights is more risky than working days. 

117. Maintainers would never be put in an unsafe situation to make the flight schedule. 

118. Maintenance performed on detachments is a lesser quality than that at homeport. 

119. W/C supervisors are allowed to run their shops without interference from M/C. 

120. | W/C supervisors feel pressure from M/C. 

121. Maintenance is only performed “by the book.” 

122. Work priorities are clearly set at the shift change maintenance meetings. 

123. Use of overdue calibration equipment is not tolerated. 

124. Maintenance personnel are often over worked. 

125. Personnel are allowed to work on aircraft or equipment when “hung-over.” 

126. Personnel are not comfortable telling supervisors that they have personal 
problems. 

127. NX works longer shifts than DX. 

128. Personnel are assigned more than one task at a time making it difficult to 


concentrate on one job. 
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129. My maintenance officer‘MCPO would understand if I did not feel comfortable 
working on an aircraft or equipment because I was having a family problem. 

130. Publications are available, understandable, and used. 

131. JIamcomfortable admitting to my supervisor that I have made a mistake. 

132. Supervisors tolerate “short-cuts.” 

133. Most maintainers would rather cover up a mistake than admit to it. 

134. Admitting to having an illness such as a cold is considered trying to “weasel out 
of work.” 

135. My work center is undermanned. 

136. My division CPO is aware of my daily workload. 

137. M/C would never put me in work on a job that could put me in an unsafe 
situation. 

138. Jam never rushed to complete a task. 

139. Much of the pressure from M/C to complete a job is perceived and not real. 

140. My supervisor is responsible for informing M/C if I feel they are 
pressuring/rushing me. 

141. All known problems are addressed or corrected prior to an aircraft being released 
“safe for flight.” 

142. Iam considered to be an important part of our maintenance department. 

143. I feel comfortable reporting someone who I consider puts me in an unsafe 
situation. 

144. Unprofessional behavior is not tolerated in the maintenance department. 

145. Maintenance management encourages me to perform maintenance “by the book.” 

146. Disciplinary problems are not considered by maintenance management to affect a 
person’s ability to perform his/her job. 

147. If an unsafe condition existed, it would be recognized by M/C, Q/A, or the W/C 
supervisor before it became a problem. 

148. All personnel in my department are physically capable of doing there jobs. 

149. Intentional violations of proper procedures are dealt with swiftly. 

150. QA is looking out for my safety. 

151. J am briefed on al possible hazards associated with performing a job prior to 
Starting that job. 

152. QA is considered to be the technical experts. 

153. QA is never pressured by maintenance leadership to sign off a gripe. 

154. My job is as important as the aircrews. 

155. My job (role) is considered to be important in achieving the overall command 
goal. 
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APPENDIX C. PROTOTYPE MCAS 


Purpose: The purpose of this survey is to try and gain valuable insight into the maintenance community’ s 
perception concerning aviation mishaps within the Navy and Marine Corps. Your participation and answers 
will be used as a guide in the Navy’s on-going efforts to lower the aviation mishap rate. 

The first fifteen questions, part I, regard biographical data; information particular to yourself. This 
information will aid in the analysis of your responses. NO attempts will be made to identify individual 
respondents or their organizations. 

Part If has 67 questions pertaining to the maintenance community. Please respond to the questions with 
the answer that most correctly reflects your honest opinion. Using a #2 pencil, completely darken each 
response. 

Thank you in advance for your participation! 


PLEASE RESPOND TO EACH ITEM. 


dle Your rank? () E-1l =8E-3 ( }E-4 =SaeE-6 ()CPO E=-7 +e) Of micer 
2s Your COmmuntey ? 
VFA (r=) ES fe) VMFA (_ ) VE es), HS ) 
VMA ( ) HC >) VP co Hes ( ) va} 
VR 7) VQ ( ) VAC VAW ( ) 
Se Your designator? ( LDO, 152X, ete )? / NEC 
a. Are you currently a department head? ( ) Yes (Sr No 
Bye Your service? ( ) USN ( ) USNR TAR ( ) SELRES ( ) Other 
Ga Your wshe te: = (7) Dx (nee. (a)? Max ( ) Other, specify 
ie Total years of service? 
8. Total years of Aviation Maintenance experience? 
- A-School graduate? ( ) Yes ( ) No (a) N/A 


10. Education level:( ) GED ( ) High School ( )College, # of years 
ads: Unit home location? ( )East coast ( )West Coast ( )Other 
iZ2. Yeurmerating 7 ( )AD/AM ( ) AE/AT ( )PR/AME ( )AO ( )Other 
dese Your age? ( ) 17-20 ()- 212s (2) 25-30 ( ) 30+ 
14. Current maintenance qualifications? 
( )Safe for Flight ( )QAR 
heb ac Ii ( )Supervisor 


Sse ( )N /A 


leche Puiss: ( ) Shore Crs) Sea 
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Part II 


l. 


My command adequately reviews and updates 
safety practices. 


The command has a dedicated program that 
targets individual training deficiencies. 


My command monitors maintainer qualifications. ( ) 


Support equipment licensing is monitored 
in this command. 


Tool Control is taken seriously at my command. 


My command recognizes individual safety 
achievement through rewards and incentives. 


Unprofessional behavior is not tolerated in the 
maintenance department. 


My command has a problem with passdown 
between shifts. 


9. My command follows established standards 


10. 


ie 


ie 


ie. 


and maintains quality control. 


CDIs/QARs are sought after positions in my 
command. 


My command temporarily restricts maintainers 
who are having personal problems. 


Based upon my command's current manning 
and assets, it is over-committed. 


My command ensures all maintainers are responsible 
and accountable for safe maintenance. 


My command ensures the uniform enforcement 
of SOPs among unit maintenance personnel. 


. Within my unit, good communication flow 


exists up and down the chain of command. 


. Coordination is conducted between M/C , W/C 


and QA prior to incorporation of TDs. 


Inspectors perform all required actions before 
sign off. 


Supervisors encourage reporting safety concerns 
without fear of retribution. 


C7 


CG) 


( ) 


20. 


Ze 


22; 


Zo 


24. 


Zoe 


ZO: 


zie 


28. 


Zo 


30. 


31. 


a2. 


33. 


34. 


52. 


36. 


a7. 


. Supervisors discourage violations of SOPs, 


or NAMP guidelines. 


To meet operational commitments, supervisors 
allow “cutting corners”. 


Supervisors manage the hazards associated with 
maintenance and flight line operations. 


Supervisors are more concerned with mission 
completion than aircraft maintenance. 


My division CPO is aware of individual 
daily workload requirements. 


Unsafe conditions are recognized and 
addressed by M/C, Q/A, or W/C supervisors. 


Supervisors communicate command safety goals, 
programs and procedures. 


Supervisors are actively involved in the safety 
program and management of safety matters. 


Supervisors set the example for compliance to 
established maintenance standards. 


Supervisors are responsive to unexpected changes 
and anticipate potential hazards. 


WI/C supervisors are respected by the maintenance 
chief/officer. 


All maintenance evolutions are properly supervised 
by qualified personnel. 


Work center supervisors , division CPOs and 
M/C work well together. 


Aircraft moves are briefed and detailed personnel 
are qualified. 


Maintainers are briefed on potential hazards 
associated with maintenance activities. 


My supervisor shields me from outside 
pressures which may affect my work. 


QARs are never pressured by the maintenance 
supervisors to sign off a gripe. 


My MO/MCPO understand if I feel uncomfortable 
performing maintenance due to personal issues. 


Maintainer staffing 1s sufficient from shift to shift. 
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38. 


oo 


40. 


4}. 


42. 


43. 


44. 


45. 


46. 


47. 


48. 


49. 


50. 


Slt 


52: 


23% 


54. 


a5. 


56. 


Maintenance control is effective in managing 
all maintenance activities. 


Multiple job assignments and collateral duties 
adversely affect maintenance. 


Personnel turnover negatively affects my 
command’s ability to operate safely. 


Violations of SOP, NAMP guidelines, or other 
procedures are common in my command. 


Proper tools and equipment are available, 
serviceable and used. 


Maintenance Control never troubleshoots 
aircraft discrepancies. 


Required publications are available, current, 
and used. 


Maintenance gripes are either corrected or 
addressed prior to flight. 


CDIs/QARs routinely monitor maintenance 
evolutions. 


My command has a reputation for quality maintenance. 
The QA division 1s respected in my command. 


Signing off PQS/JQRs/PARs Is taken seriously 
and not gun decked. 


Maintenance quality on detachments is the same (_ ) 
as that in homeport. 


Day and Night Check have equal workload and 
are equally stressful/fatiguing. 


QARs are viewed as helpful, and QA 1s 
not “feared” in my command. 


Peer influence discourages violations of SOP, 
NAMP guidelines or other procedures. 


Personnel are uncomfortable telling supervisors 
about personal problems including illness. 


I am provided adequate resources (time, personnel 
and equipment) to accomplish my job. 


In my command, we believe safety is an integral 
part of all maintenance and flight line operations. 
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Di 


58. 


59) 


60. 


61. 


G2. 


63. 


G5) 


66. 


67. 


I feel I get all information (internal and external) 
required to perform my job safely. 


Individuals feel free to report safety violations, 
unsafe performance, or other unsafe behavior. 


My command uses safety staff to manage 
personnel at risk. 


Our command climate promotes safe maintenance 
and flight operations. 


Safety decisions are made at the proper 
command levels. 


Safety is part of maintenance planning, and 
additional training/support is provided as needed. 


Maintainers are never purposely put in an 
unsafe situation to meet the flight schedule. 


Safety education and training in my command are 
comprehensive and effective. 


The safety department is respected by 
supervisors and maintainers. 


Maintenance Safety Petty Officer is a sought after 
billet in my command. 


The command uses medical staff to manage 
occupational hazards and personnel at risk. 
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APPENDIX D. COMPONENT SURVEY DESCRIPTIVE RESULTS 


Patrol Combat Support Combined 
a Squadron Squadron Squadron Squadrons 
| i Total] Avg |StdDev|Total| Avg [StdDeyTotal] Avg |StdDev|Total] Avg |StdDev 
| 2 | 108 {| 3.86 | 0.942 | 93 | 3.90 [0.835] 67 | 3.72 | 1.070 | 268 | 3.84 | 0.940 | 
| 3 107 4.44 | 0.881 | 92 | 4.29 [0.749] 66 | 4.27 | 0.755 | 265| 4.35 | 0.849 | 
| 46 105] 4.24 | 1.020 90 | 4.18 10.680} 65 | 4.20 | 0.775 | 260| 4.21 | 0.960 | 
| 59 | 104] 3.92 | 1.050 | 90 | 3.91 | 0.802] 67 | 3.73 | 1.109 | 261] 3.87 | 1.088_ 
| 67__—« 105] 3.30_| 1.050 | 91 | 3.49 | 1.004} 66 | 3.08 | 1.154 | 262] 3.31 | 1.119 


MOSE Process Auditing Component 
Squadron Squadron Squadron Squadrons 

|__—{Totall Avg |StdDev Total] Avg StdDeyTotall Avg |StdDevTotall Avg |StdDev 
| 6 —«f 106] =3.60 | 0.976| 93 | 3.83 | 3.828] 67 | 3.90 | 1.103| 266] 3.78 | 1.120 | 
| 19 =| 105{ 3.40 | 1.350] 92 | 4.11 [0.883] 66 | 4.03 | 0.960| 263] 3.85 | 1.216 | 
| 36 101] 3.38 | 1.120] 89 | 3.28 | 0.988] 64 | 3.17 | 1.121] 254] 3.28 | 1.187 | 
| 41 =f 105] 2.45 | 1.180] 93 | 2.11 | 1.098] 66 | 1.94 | 1.006] 264] 2.16 | 1.145 
es ci 8105) | 3168 1.090 ||o2)i|| s'62) (01959166 is Semi|| 1099"|"2ealmeiGall! 1.1107 | 
| 58 105] 4.23 | 0.990| 92 | 4.35 |0.702) 66 | 4.05 | 1.143] 263] 4.21 | 1.044 
| 60 105} 4.38 | 0.810] 92 | 4.42 | 0.683} 67 | 4.28 | 0.901 | 264 


MOSE Reward System and Safety Culture Component 


Fleet Support Patrol Combined 
Squadron Squadron Squadron Squadrons 
otal StdDev| Total tdDe 
0.67 




















| 3.54 | 1.300 | 90 | 3.89 | 
. | 89 | 


103] 3.84 | 1.230 | 91 | 4.01 [0.960] 66 | 3.88 _ 





MOSE Quality Component 
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Fleet Support Patrol Combat Support Combined 


=_ Squadron Squadron Squadron Squadrons 
a 






Total Total] Avg StdDey Total Total 

103 0.918 
| 12} 105} 2.93 | 1.120] 92 | 3.15 [0.983] 66 | 3.00 | 1.123 | 263] 3.03 | 1.116 | 
| 21 | 106 | 3.94 | 1.000 | 92 | 3.89 |0.805| 66 | 3.98 | 0.903 | 264 | 3.94 | 0.969 | 
| 22} 107 | 2.32 | 1.290] 92 | 2.03 [0.966] 66 | 2.15 | 1.056 | 265] 2.17 | 1.135 | 
| 23) 105} 3.80 | 1.100 | 90 | 3.73 |0.981] 63 | 3.65 | 1.109 | 258] 3.73 | 1.125 | 
| 24 | 107} 4.33 _| 0.760 | 92 | 4.30 [0.752] 66 | 4.21 | 0.814 | 265] 4.28 | 0.813 | 
| 40 105} 2.92 | 1.270 | 91 | 2.93 [0.892] 66 | 2.86 | 1.201 | 262 | 2.91 | 1.165 | 
| 51 | 103} 2.60_| 1.220 | 87 | 3.21 10.904] 66 | 2.68 | 1.166 | 256] 2.83 | 1.189 | 
| 63 104] 3.80 | 1.190 | 92 | 4.07 | 1.003] 66 | 4.12 | 0.969 | 262} 3.99 | 1.174 | 


MOSE Risk Management Component 


Squadron Squadron Squadron 
| Tot | Avg | SD_| Tot| Avg | SD | Tot} Avg | SD_| Tot] Avg | SD_ 
| 14 | 106} 3.98 | 0.976 | 92 | 4.03 [0.870] 66 | 4.12 | 0.775 | 264 | 4.04 | 0.954 | 
| 25 | 106} 4.08 | 0.950 | 92 | 4.12 [0.782] 66 | 3.98 | 0.920 | 264| 4.06 | 0.951 | 
| 26] 106 | 3.98 | 1.000 | 92 | 4.18 |0.662| 66 | 4.05 | 0.919 | 264 | 4.07 | 0.939 | 
| 27 107 | 3.95 | 0.960| 91 | 4.14 [0.754] 66 | 3.97 | 0.911 | 264 | 4.02 | 0.913 | 
| 28 | 106 | 3.97 | 0.950] 92 | 4.01 | 0.791] 66 | 3.88 | 0.851 | 264 | 3.95 | 0.933 _ 
| 29 | 107 | 3.79 | 1.070 | 91 | 3.45 | 1.213] 66 | 3.83 | 1.090 | 264 | 3.69 | 1.155 | 
| 30 106 | 3.85 | 1.090] 92 | 4.15 [0.725] 66 | 4.08 | 0.966 | 264 | 4.03 | 1.011 | 
| 38 | 105 | 3.92 | 1.090] 93 | 3.82 | 0.988] 66 | 3.77 | 1.049 | 264 | 3.84 | 1.114 | 
| 39 104 | 3.30_| 1.200] 92 | 3.10 ]0.973| 66 | 3.36 | 1.159 | 262 | 3.25 | 1.175 | 
| 64 | 105] 4.07 | 0.960] 92 | 4.10 | 0.757] 67 | 4.12 | 0.879 | 264 | 4.09 | 0.968 | 
| 66 | 102} 3.03 | 1.100] 90 | 3.21 } 0.966] 67 | 3.10 | 1.032 | 259] 3.12 | 1.126 | 


MOSE Command and Control Component 


ae ie || set A elle | Sieneee, 
Squadron Squadron Squadron Squadrons 
| Cf Totall Avg |StdDev|Total] Avg SBtdDeyTotal] Avg |StdDev|Total] Avg |StdDev 
| CS C102 2.78 | 0.953] 92 | 2.38 [1.137] 67 | 249 | 1.146 | 261) 255 | 1.237, 
| 16] 103 3.88 | 0.953] 91 | 3.65 [0.970] 67 | 3.73 | 0.827 | 261] 3.75 | 1.055_ 
| 31] 106 | 3.78 | 1.100] 92 | 3.60 | 1.070] 66 | 3.61 | 1.149 | 264] 3.66 | 1.141 
4 
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| 32] 105 4.00 | 1.050] 92 | 4.40 [0.696] 66 | 3.91 | 1.092 | 263] 4.10 | 1.060_ 
92] 65 | 4.00 | 0.952 | 260 | 3.99 | 1.053 
| 64 | 2.91 | 1.019 | 254 | 3.01 | 1.092 | 
| 66 | 3.45 | 1.126 | 260] 3.55 | 1.228 | 
| 66 _ . 
| 66 _ . . 
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MOSE Communication / Relationship Component 
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APPENDIX E. SUGGESTED QUESTIONS MODIFICATIONS 


9. My command adequately reviews and updates safety practices. 

1. My command follows established standards and maintains quality control. 

13. My command ensures all maintainers are responsible and accountable for safe 
maintenance. 


New question: My command adequately reviews and updates safety practices, follows 
established standards and maintains quality control, while ensuring that all maintainers 
are responsible and accountable for safe maintenance. 


3. | Mycommand monitors maintainer qualifications. 
4. Support equipment licensing is monitored in this command. 


New question: My command monitors maintainer qualifications and support equipment 
licensing. 


57. I feel I get all information (internal and external) required to perform my job safely. 
58. Individuals feel free to report safety violations, unsafe performance, or other unsafe 
behavior. 


New question: | feel I get all information required to perform my job safely, and feel free 
to report safety violations, unsafe performance or other unsafe behavior. 


30. All maintenance evolutions are properly supervised by qualified personnel. 
33. Maintainers are briefed on potential hazards associated with maintenance activitites 


New Question: Qualified personnel properly supervise all maintenance evolutions and 
maintainers are briefed on the potential hazards associated with maintenance activities. 


42. Proper tools and equipment are available, serviceable and used. 
55. Iam provided adequate resources (time, personnel, and equipment) to accomplish my 
job. 


New question: Proper tools and equipment are available, serviceable and used an I am 
provided adequate resources (time, personnel) to accomplish my job. 


49. Signing off PQS/JQRs/PARs is taken seriously and not gun decked. 
50. Maintenance quality on detachments is the same as that in homeport. 


New Question: Signing off PQS/JQRs/PARs is taken seriously, not gun decked and 
maintenance quality is as high on detachments as it is in homeport. 
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2. The command has a dedicated program that targets individual training deficiencies. 
14. My command ensures the uniform enforcement of SOPs among maintenance 
personnel. 


New question: My command has a dedicated program that targets individual training 
deficiencies and ensures the uniform enforcement of SOPs among maintenance 
personnel. 


37. Maintainer staffing is sufficient from shift to shift. 
51. Day and night check have equal workload and are equally stressful / fatiguing. 


New question: Maintainer staffing is sufficient, is equally worked and is equally stressed 
/ fatigued from shift to shift. 


16. Coordination is conducted between M/C, W/C, and QA prior to incorporation of TDs. 
38. Maintenance control is effective in managing all maintenance activities. 


New question: Maintenance control is effective in managing all maintenance activities, 
coordinating between M/C, W/C, and QA prior to the incorporations of TDs. 


20. To meet operational commitments, supervisors allow cutting comers. 
22. Supervisors are more concerned with mission completion than aircraft maintenance. 


New question: Supervisors are more concemed with proper aircraft maintenance than 
mission completion and do not allow cutting corners to meet operational commitments. 


64 Safety education and training in my command are comprehensive and effective. 
65. The safety department is respected by supervisors and maintainers. 


New question: Safety education and training in my command are comprehensive and 


effective and the safety department is respected by the supervisors and maintainers. 


52. QARs are viewed as helpful, and QA is not “feared” in my command. 
56. In my command, we believe safety is an integral part of all maintenance and flight 
line operations. 


New question: Safety is an integral part of this command’s maintenance planning/flight 
line operations, where QARs are helpful and the QA division is not “feared”. 


19. Supervisors discourage violations of SOPs, or NAMP guidelines. 
53. Peer influence discourages violations of SOP, NAMP guidelines or other procedures. 


New question: Violations of SOP, NAMP guidelines or other procedures are discouraged 
in this command. 
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5. Tool control is taken seriously in my command. 
47. My command has a reputation for quality maintenance. 


New question: My command has a reputation for quality maintenance and tool control is 
taken seriously. 


18. Supervisors encourage reporting safety concerns without fear of retribution. 

21. Supervisors manage the hazards associated with maintenance and flight line 
operations. 

24. Unsafe conditions are recognized and addressed by M/C, Q/A, or W/C supervisors. 


New question: Safety concerns or unsafe hazards associated with maintenance and flight 
line operations can be reported without a fear of retribution knowing that the W/C, Q/A, 
or M/C supervisors will address and manage them for proper corrections. 


11. My command temporarily restricts maintainers who are having personal problems. 
36. My MO/MCPO understand if I feel uncomfortable performing maintenance due to 
personal issues 
67. The command uses medical staff to manage occupational hazards and personnel at 
risk. 


New question: Medical staff is used to help identify, manage, and temporarily restrict 
personnel with personal issues and those who pose a risk to safe maintenance in this 
command. 


10. CDIs and QARs are sought after positions in my command. 
66. Maintenance Safety Petty Officer is a sought after billet in my command. 


New question: QARs/ CDIs and Maintenance Safety Petty Officer are sought after 
billets in my command. 


45. Maintenance gripes are either corrected or addressed prior to flight. 

46. CDIs/QARs routinely monitor maintenance evolutions. 

48. The QA division is respected in my command. 

New question: The QA division is respected and CDIs / QARs routinely monitor 
maintenance evolutions ensuring that maintenance gripes are either corrected or 
addressed prior to flight. 


Question 8: 


Current: My command has a problem with passdown between shifts. 
New question: My command does not have a problem with passdown between shifts. 


(e 


Question 12: 


Current: Based upon my command’s current manning and assets, it is over-committed. 
New question: Based upon my command’s current manning and assets, it is not over- 
committed. 


Question 20 


Current: To meet operational commitments, supervisors allow “cutting comers.” 
New question: Supervisors do not allow “cutting corners” to meet operational 
commitments. 


Question 22 


Current: Supervisors are more concerned with mission completion than aircraft 
maintenance. 

New question: Supervisors are more concerned with proper aircraft maintenance than 
mission completion. 


Question 39 


Current: Multiple job assignments and collateral duties adversely affect maintenance. 
New question: Multiple job assignments and collateral duties do not adversely affect 
maintenance. 


Question 40 


Current: Personnel turnover negatively affects my command’s ability to operate safely. 
New question: Personnel turnover does not affect my command’s ability to operate 
safely. 


Question 41 


Current: Violations of SOP, NAMP guidelines, or procedures are common in my 
command. ; 
New question: Violations of SOP, NAMP guidelines, or procedures are not common in 
my command. 
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Question 54 


Current: Personnel are uncomfortable telling supervisors about personal problems 


including illness. 
New question: Personnel are comfortable telling supervisors about personal problems 


including illness. 
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APPENDIX F. FINALIZED MCAS 


Purpose: The purpose of this survey is to try and gain valuable insight into the maintenance community’s 
perception concerning aviation mishaps within the Navy and Marine Corps. Your participation and answers 
will be used as a guide in the Navy’s on-going efforts to lower the aviation mishap rate. 

The first fifteen questions, part I, regard biographical data; information particular to yourself. This 
information will aid in the analysis of your responses. NO attempts will be made to identify individual 
respondents or their organizations. 

Part II has 35 questions pertaining to the maintenance community. Please respond to the questions with 
the answer that most correctly reflects your honest opinion. Using a #2 pencil, completely darken each 
response. 

Thank you in advance for your participation! 


PLEASE RESPOND TO EACH ITEM. 


ie Your rank? ( ) E-1 - E-3 ( )E-4 - E-6 ( )CPO E-7 + ( )Officer 

oe Your community? 
VFA ( ) HS Aca) VMFA (_ ) VE (gag) Siig } 
VMA (ad HC ( ) VP ( ) HCS ( ) Mel ) 
VR ) VQ 3) VAQ ( ) VAW i 

a Your designator? ( LDO, 152X, etc )? 7 NEC 

Ae Are you currently a department head? ( ) Yes ( JeNo 

De Your service? (>) UsN (eieUSNR BAR jo) SELRES@( jmOther 

6. Your shift? (bx in) NX ( ) Marae ( ) Other 

ls Total years of service? 

or Total years of Aviation Maintenance experience? 

9. A-School graduate? ( ) Yes ( ) No ( ) N/A 


lil.  Edtication Tevel:( ) GED (3 Hromesecncom™ ) College, #V¥rs 
iZ eee Unae e home Location? ( )Bast coast ( jWest Coast ( jOther 


2 Youre raring ( )AD/AM ( ) AE/AT ( )PR/AME ( )AO ( )Other 

ie Your age? a 7-20 (oa)s 225 ( ) 25-30 (3,0. 

14. Current maintenance qualifications? 
( )Safe for Flight ( )QAR 
(eames ( )Supervisor 
( ) SPO ( )N /A 


eee Boh anya. } Shore Ce) Sea 


st 


Part II 


ll 


10. 


Lie 


12 


[3. 


My command has a dedicated program that 

targets individual training deficiencies and ensures 
the uniform enforcement of SOPs among maintenance 
personnel. 


My command monitors maintainer qualifications 
and support equipment licensing. 


My command has a reputation for quality 
maintenance and tool control is taken seriously. 


Unprofessional behavior is not tolerated in the 
maintenance department. 


My command has a problem with passdown 
between shifts. 


My command adequately reviews and updates 

safety practices, follows established standards and 
maintains quality control, ensuring that all maintainers 
are responsible and accountable for safe maintenance. 


QARs/ CDIs and Maintenance Safety Petty Officer are 
sought after billets in my command. 


. Medical and safety staff are used to help identify, 


manage, and temporarily restrict personnel with personal 
issues and those who pose a risk to safe maintenance in 
this command. 


Based upon my command’s current manning 
and assets, it is not over-committed. 


Within my unit, good communication flow 
exists up and down the chain of command. 


Maintenance control 1s effective in managing all 
maintenance activities, coordinating between M/C, 
WIC, and QA prior to the incorporations of TDs. 


Inspectors perform all required actions before 
sign off. 


Safety concerns or unsafe hazards associated 

with maintenance/flight line operations can be 

reported without fear of retribution knowing that the 
WIC, Q/A, or M/C supervisors will address and manage 
them for proper corrections. 


Violations of SOP, NAMP guidelines or other 
procedures are discouraged in this command. 
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16. 


WE 


18. 


ie 


20. 


2K 


22, 


ae 


24. 


257 


26. 


De 


Zoe 


Supervisors are more concerned with proper aircraft 
maintenance than mission completion and do not 
allow cutting corners to meet operational commitments. 


My supervisors are aware of individual 
daily workload requirements and recognize safety 
achievements through rewards and incentives. 


Supervisors communicate command safety goals, 
programs and procedures. 


WI/C supervisors are respected by the maintenance 
chief/officer. 


Qualified personnel properly supervise all 
maintenance evolutions and maintainers are 
briefed on the potential hazards associated with 
maintenance activities. 


My supervisor shields me from outside 
pressures which may affect my work. 


QARs are never pressured by the maintenance 
supervisors to sign off a gripe. 


Maintainer staffing is sufficient, is equally worked and 
is equally stressed / fatigued from shift to shift. 


Multiple job assignments and collateral duties 
do not adversely affect maintenance. 


Personnel turnover does not affect my 
command’s ability to operate safely. 


Violations of SOP, NAMP guidelines, or other 
procedures are not common in my command. 


Proper tools and equipment are available, 
serviceable and used and I am provided adequate 
resources (time, personnel) to accomplish my job. 


Maintenance Control never troubleshoots 
aircraft discrepancies. 


Required publications are available, current, 
and used. 


29. The QA division is respected and CDIs / QARs 


30. 


routinely monitor maintenance evolutions ensuring 
that maintenance gripes are either corrected or 
addressed prior to flight. 


Signing off PQS/JQRs/PARs is taken seriously, 
not gun decked and maintenance quality is as high 
on detachments as it is in homeport. 


a 


on 


S72. 


oe 


34. 


as 


Safety 1s an integral part of this command's 
maintenance planning/flight line operations, where 
QARs are helpful and the QA division is not “feared”. 


Personnel are comfortable telling supervisors 
about personal problems including illness. 


I feel I get all information (internal and external) 
required to perform my job safely, and feel free to 
report safety violations, unsafe performance or other 
unsafe behavior. 


Maintainers are never purposely put in an 
unsafe situation to meet the flight schedule. 


Safety education and training in my command are 


comprehensive and effective and the safety department 
1s respected by the supervisors and maintainers. 
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